A Generative Approach to Joint Modeling of Quantitative and Qualitative Responses

In many scientific areas, data with quantitative and qualitative (QQ) responses are commonly encountered with a large number of predictors. By exploring the association between QQ responses, existing approaches often consider a joint model of QQ responses given the predictor variables. However, the dependency among predictive variables also provides useful information for modeling QQ responses. In this work, we propose a generative approach to model the joint distribution of the QQ responses and predictors. The proposed generative model provides efficient parameter estimation under a penalized likelihood framework. It achieves accurate classification for qualitative response and accurate prediction for quantitative response with efficient computation. Because of the generative approach framework, the asymptotic optimality of classification and prediction of the proposed method can be established under some regularity conditions. The performance of the proposed method is examined through simulations and real case studies in material science and genetics.

Introduction
Analyzing data with heterogeneous types of responses has been an important topic with broad applications. Such heterogeneous data often involve both quantitative and qualitative (QQ) responses. For example, Klein et al. (2019) described a human health study examining the risk factors of adverse birth outcomes, which contains a qualitative response "presence/absence of low birth weight" and a quantitative response "gestational age". In material science, the properties of a material are often characterized by QQ measures. As shown in the case study of Section 5 on Heusler compounds, two metrics, the mixing enthalpy (quantitative) and the global stability based on hull energy (qualitative) are used to determine the thermodynamic stability of a full Heusler compound. The QQ responses is a special case of "mixed outcomes" in the literature. In this paper, we focus on two types of mixed responses: the quantitative continuous response and the multi-class qualitative response.
In the literature of mixed outcomes, particularly on QQ responses, it has been known that overlooking the relationship between the QQ responses is inappropriate. Researches on the joint model of the QQ responses include early ones such as Fitzmaurice and Laird (1995); Moustaki and Knott (2000); Dunson (2000); Gueorguieva and Agresti (2001); Dunson (2003) and recent ones such as Deng and Jin (2015); Kürüm et al. (2016); Kang et al. (2018); Amini et al. (2018); Klein et al. (2019). These works mostly consider a joint regression model conditioned on the predictor variables. Based on our best knowledge, we can group them into two categories.
The first group of methods considers a conditional regression model, in which one type of the QQ responses is treated as the response of the model, and the other type is treated as the regressor. For instance, Fitzmaurice and Laird (1997) introduced a marginal regression model of quantitative response conditioned on the qualitative response. Song et al. (2009) used Gaussian copulas to integrate separate one-dimensional generalized linear models into a joint regression model for mixed outcomes. Lin et al. (2010) developed a conditional mixedeffects model to analyze clustered data containing QQ responses. These methods are suitable for data with a small number of predictor variables. To handle the high-dimensional input variables, Deng and Jin (2015) proposed a conditional model that encourages model sparsity through a constrained likelihood estimation. However, inferences and asymptotic properties of their method have not been explored due to the complicated constrained likelihood estimation. Kang et al. (2018) considered a Bayesian estimation for the conditional model of Deng and Jin (2015) to obtain proper inferences of model parameters. Nevertheless, their work is not designated for studying the asymptotic properties of the proposed estimator.
The second group of methods considers a continuous latent variable for the qualitative response, and then jointly models the latent variable and the quantitative response (Sammel et al., 1997;Dunson and Herring, 2005;Bello et al., 2012). For example, Gueorguieva and Agresti (2001) studied a probit model with a latent variable and developed a Monte Carlo expectationconditional maximization algorithm for parameter estimation. Klein et al. (2019) introduced the idea of latent variable into the framework of copula regressions, constructing a latent continuous representation of binary regression models. However, the use of latent variables often involves considerable computation in the parameter estimation. It also makes the investigation of theoretical properties difficult. Moreover, most of these works focus on the binary qualitative response and their model assumptions may not be easily extended to the multi-class qualitative response cases.
In this work, we propose a novel approach to jointly model the QQ responses based on the generative approach. The proposed generative model considers the joint distribution of the high-dimensional input variables, the quantitative responses, and the multi-class qualitative response. It is a very unique and different perspective from the existing literature and also brings advantages in both theoretical and computational aspects. The proposed method can accommodate multi-class qualitative response and multivariate quantitative responses with attractive theoretical properties. We call the proposed method GAQQ, a Generative Approach for QQ responses.
The key contributions of this work are summarized as follows. First, based on the generative model framework, we are able to establish the asymptotic properties of the proposed estimators with respect to both the classification accuracy of the qualitative response and the prediction accuracy of the quantitative response under some regularity conditions. Such conditions are commonly used in the regularized estimation framework (Shao et al., 2011;Zhao and Yu, 2006). The classification of the qualitative response enjoys the asymptotic optimality of the resulting linear discriminate classification rule. The mean squared error (MSE) of prediction for the quantitative response is as good as the optimal prediction under the Bayes risk. Second, an efficient procedure for parameter estimation is developed via the regularized log-likelihood function of the joint distribution of input variables and QQ responses. Specifically, we impose regularization on both the mean differences and the covariance matrix from the joint distribution to achieve sparsity for high-dimensional predictor variables. Third, the use of the generative approach leads to an effective prediction procedure by inferring the conditional distribution of QQ responses conditioned on the predictor variables. That is, the quantitative response is predicted through the property of conditional multivariate normal distribution, and the linear discriminant analysis (LDA) is employed for classification of the qualitative response. Fourth, the proposed generative model allows the parameters related to QQ responses to be mutually learned from each other, which is different from existing methods in which only modeling one type of QQ responses attempts to benefit from the information of the other type of QQ responses.
The remainder of this paper is organized as follows. Section 2 details the proposed method. The main theoretical results are presented in Section 3. Simulation and real data analysis are conducted in Sections 4 and 5, respectively. Section 6 concludes this work with some discussion. Technical proofs are in the Appendix.

The Proposed Model
Suppose that the variables of interest are denoted by (X, y, Z) where X = (X 1 , . . . , X p−1 ) ′ is a (p − 1) dimensional vector of predictor variables, y is a quantitative response variable and Z ∈ {1, 2} is a qualitative response variable. From a generative modeling perspective, we consider the data generation mechanism as p(X, y, Z) = p(X, y|Z)p(Z), indicating that data are from two classes G 1 and G 2 under (X, y)|Z. Assume that W = (X ′ , y) ′ follows multivariate norm distributions with different means for two classes, but sharing the same covariance matrix as follows (1) Denote the observed data w 1 , · · · , w n 1 , w n 1 +1 , · · · , w n 1 +n 2 with the first n 1 observations from G 1 and the rest n 2 observations from G 2 , where w i = (x ′ i , y i ) ′ , i = 1, 2, . . . , n 1 + n 2 . Let n = n 1 + n 2 . The log-likelihood function can be written as up to some constant, where C = Σ −1 is the inverse covariance matrix. Let π 1 and π 2 be the prior probability of w belonging to classes G 1 and G 2 , respectively. Hence, the LDA assigns where δ = µ 1 − µ 2 . Otherwise, w is classified to G 2 . The estimates of π 1 and π 2 are the empirical proportions of data from each class. The parameters µ 1 , µ 2 and C can be estimated by maximizing the log-likelihood function of (2).
For high-dimensional data when p ≥ n, the regularization is often needed to ensure the proper estimation of inverse covariance matrix C and mean difference δ. We thus propose to penalize C = (c ij ) 1≤i,j≤p and δ simultaneously, resulting in the following optimization where ||C|| 1 = i =j |c ij |, and |α| 1 = i |α i | with α i being the ith entry of vector α. Here λ 1 ≥ 0 and λ 2 ≥ 0 are two tuning parameters. By applying such regularization, the proposed model can encourage the sparse structures in C and δ at the same time. Note that similar spirits of regularizing both C and δ are used in several works on the LDA (Shao et al., 2011;Cai and Liu, 2012).
To estimate the parameters, we develop an iterative procedure to solve the sub-optimization problem with respect to C and δ respectively. Define δ 2 = (µ 1 − µ 2 )/2 as well as γ = (µ 1 + µ 2 )/2, then accordingly we have µ 1 = δ 2 + γ and µ 2 = γ − δ 2 . As a result, the optimization problem (4) is re-written as It is thus easy to obtain the maximum likelihood estimate of γ from (5) aŝ wherew = 1 n n i=1 w i is the overall mean. Then pluggingγ back into (5) yields (δ 2 ,Ĉ) = arg min In this manner, solving the optimization problem (4) is equivalent to solving the optimization problem (7). Next, we show that (7) can be decomposed as a graphical lasso model (Glasso) (Yuan and Lin, 2007;Deng and Yuan, 2009) in terms of C and a Lasso regression (Tibshirani, 1996) in terms of δ 2 with the other parameter fixed, such that these two parameters can be estimated iteratively. To be more precise, for a given value of δ 2 , the minimization problem (7) with respect to C is min C −n ln |C| + tr(CS) + λ 1 ||C|| 1 , It has the same form as the graphical lasso, which has been extensively studied in literature by Yuan and Lin (2007);Friedman et al. (2008); Lam and Fan (2009);Raskutti et al. (2008); Liu et al. (2020), and many others. On the other hand, when the inverse covariance matrix C is fixed, the minimization problem (7) regarding δ 2 becomes min δ 2 i∈G 1 which is equivalent to min δ 2 whereỹ = 1 2n 1 n 2 C 1/2 (n 2 i∈G 1 w i − n 1 i∈G 2 w i ). A detailed derivation of (10) from (9) is provided in the Appendix. We solve the minimization problem (10) by the Lasso technique.
Consequently, solving the complicated optimization problem (7) is decomposed to the simple tasks of iteratively solving a Glasso estimate for C and a Lasso estimate for δ 2 until both of them are converged. We summarize the above estimation procedure for the proposed model in Algorithm 1.
Step 0: Set an initial value of δ 2 .
Note that there are two tuning parameters λ 1 and λ 2 in the optimization problem (7).
To choose their optimal values, we minimize a BIC-type criterion proposed by Wang et al. (2007) as where v(δ 2 ) and v(Ĉ) stand for the number of nonzero entries in the estimatesδ 2 andĈ, respectively. This criterion enjoys consistency properties and has been commonly used in literature (Zou and Zhang, 2009;Lv and Fan, 2009;Armagan et al., 2013).

Model Prediction
In this section, we demonstrate how to conduct model prediction by the proposed method.
For convenience, let us write where µ 1X and µ 2X are p−1 dimensional vectors representing the means of variable X for two classes, and Σ X is the (p−1)×(p−1) covariance matrix of X. The estimatesμ 1 ,μ 2 ,Ĉ,Σ can be partitioned accordingly. From model assumption (1) as well as the property of multivariate Therefore, the prediction for the quantitative variable y from a new observation x iŝ Note thatΣ ′ Xy whereĉ 2 y is a scalar, implying that the sparsity ofĈ Xy will lead to the sparse model for the prediction of y.
On the other hand, the prediction for the qualitative variable Z by the proposed model is naturally based on the estimated LDA classification rule of (3) as From Equations (11) and (12), however, we note that the prediction of one response variable depends on the information of the other. To address this issue, we propose to calculate two candidate values of y for a new observation x by Equation (11) for two different classes, denoted byŷ 1 andŷ 2 . Then the conditional probability densities p(W = (x ′ ,ŷ 1 ) ′ |G 1 ) and p(W = (x ′ ,ŷ 2 ) ′ |G 2 ) can be estimated via the density functions of N(μ 1 ,Σ) and N(μ 2 ,Σ).
Denote such two values asp 1 andp 2 . The prediction of y at this new observation is then obtained asŷ k corresponding to the larger value ofπ kpk , k = 1, 2. To express it clearly, we describe the above steps of the model prediction in Algorithm 2 for a new observation x.
It is seen that in Algorithm 2, we obtain the prediction of y first, and then predict Z with the value ofŷ. One would argue that it is not a unique way of making predictions on QQ responses, as we may also predict Z first and then variable y. The following proposition provides an interesting insight into this issue. Proposition 1 implies that we can predict the response variable Z by simply comparing values ofπ kpk instead of employing LDA. Therefore, the order of which response variable to be predicted first is not a concern. Actually, the Step 2a and Step 2b are equivalent to the following Step 2 as Step 2: Ifπ 1p1 >π 2p2 , letŷ =ŷ 1 andẐ = 1; otherwise letŷ =ŷ 2 andẐ = 2.

Extension to Multi-Class Qualitative Response
The proposed generative modeling approach also has the advantage to enable the GAQQ to deal with the qualitative response with multiple classes, i.e., the qualitative variable Z ∈ {1, 2, . . . , K}. In such cases, the GAQQ method is extended and expressed as G k : Based on a baseline class G 1 , we regularize on the difference between means through µ k − µ 1 for k = 2, 3, . . . , K. The objective function is thus formulated as The subsequent derivation follows similar steps as that described in Section 2.1. With a little abuse of notation, let Kδ k = µ k − µ 1 for k = 1, 2, . . . , K and Kγ = K k=1 µ k , then we have µ k = γ − K g=2 δ g + Kδ k , k = 1, 2, . . . , K. As a result, the optimization problem (13) can be re-written as min (δ 2 ,...,δ K ,γ,C) Let n k represent the number of observations belonging to class G k . The maximum likelihood estimator of γ from (14) isγ =w + K g=2 δ g − K n K g=2 n g δ g . Consequently, the optimization problem (14) becomes (15) can be decomposed as one Glasso problem min C −n ln |C| + tr(CS) + λ 1 ||C|| 1 , and min (δ 2 ,...,δ K ) The optimization problem (16) is equivalent to the following K − 1 Lasso regressions sepa- The detailed derivation from (16) to (17) is provided in the Appendix. Therefore, the parameters δ k and C can be solved iteratively until convergence following the spirit of Algorithm 1. The optimal values of tuning parameters are chosen by the BIC-type criterion extended for the multi-class problem as where v(δ) represents the number of nonzero entries in all the estimatesδ k .
For a new observation x, the quantitative response y is predicted, similarly as in Algo- The class label is estimated asẐ = arg max kπ kpk , or equivalently by the LDA rule aŝ

Theoretical Properties
In this section, we will investigate the asymptotic optimality of the classification rule by the proposed GAQQ method in Theorem 1 to Theorem 3. The asymptotic consistency properties of the prediction of y by the GAQQ method are established in Theorem 4. For the proposed classification rule, we first establish the theoretical results for the multi-class problem and then provide a thorough discussion of the two-class case. We use the same definition of asymptotic optimality for a classification rule as defined in Shao et al. (2011).
Denote by R Bayes and R P ROP (T ) the Bayes error and the conditional misclassification rate of the proposed rule, where T denotes the training samples. The asymptotic optimality for a classification rule is defined as follows.

Definition 1. Let T be a classification rule with conditional misclassification rate R T (T ),
given the training samples T .
(1) T is asymptotically optimal if R T (T )/R Bayes P → 1. ( Note that if lim n→∞ R Bayes > 0, then the asymptotically sub-optimality is the same as the asymptotically optimality. To facilitate the construction of theoretical results, we need to introduce some notation and make assumptions on the true model. Define the true values of µ k , Σ, C and δ k as µ 0 k , Σ 0 , C 0 and δ 0 kX is a p − 1 dimensional vector representing the true mean difference of variable X between classes G 1 and G k . Denote the true inverse covariance matrix of variable X by C 0 , which is the set containing location indices of the nonzero entries in δ 0 k . Lets k be the cardinalities of set S δ k . Define s k =s k if δ 0 ky = 0; otherwise s k =s k − 1. That is, s k is the number of nonzero entries of δ 0 kX . Additionally, we use the same sparsity measure on Σ 0 = (σ 0 ij ) 1≤i,j≤p as in Bickel and Levina (2008), which is S h;p = max i≤p p j=1 |σ 0 ij | h where 0 ≤ h < 1 and 0 0 is defined to be 0. Hence firstly, S 0;p equals the maximum of the numbers of nonzero entries in each row of the matrix Σ 0 . In this case, a smaller value of S 0;p compared with p implies a sparse structure in matrix Σ 0 . Secondly, if S h;p is smaller than p for 0 < h < 1, it indicates that many entries of matrix Σ 0 are very small. Moreover, we assume the following regularity conditions.
where λ min (C 0 ) and λ max (C 0 ) are the minimum and maximum eigenvalues of matrix , and J c represents the complement set of J.
• (C4) Irrepresentable condition: without loss of generality, write δ 0 where 1 is a p −s k dimensional unit vector, and the inequality holds element-wise.
By conditions (C1) and (C2), Rothman et al. (2008) and Lam and Fan (2009) derived the convergence rate of Glasso estimate. We thus have where d n = S h;p ( log p n ) (1−h)/2 , and A is the matrix spectral norm defined as the squared root of the maximum eigenvalue of matrix A ′ A. The conditions (C2) and (C3) are used in Bühlmann and Van De Geer (2011) to study the theoretical property of Lasso estimate, and we have where b Under conditions (C4) and (C5), Zhao and Yu (2006) showed that the Lasso estimate is model selection consistency, which will be used for investigating δ k in (17). Condition (C6) requires that all the classes should be separated from each other. Also note that condition (C6) is equivalent to that ∆ k is bounded away from 0. The condition (C7) guarantees a balanced sample size for each class, which is commonly used in literature to bound the term log π k π 1 in the LDA rule for establishing the properties of classification rules. Based on the above results, we present the following theories on the consistency of the classification rule by the proposed method.

Theorem 1. Assume that conditions (C1) -(C7) hold, and
Then the proposed rule for the multi-class problem is asymptotically sub-optimal if either one of the following two conditions is satisfied Theorem 1 establishes the sub-optimality property of the proposed classification rule for the multi-class problem. In the case of two-class problem, the Bayes error can be expressed in a closed form of R Bayes = Φ(−∆ 2 /2) when the data are from normal distribution, where Φ represents the cumulative distribution function of N(0, 1), and ). Accordingly, in Theorems 2 and 3, we can compute the convergence rate of the proposed rule for the two-class problem, and subsequently investigate its properties.
Theorem 3. Assuming all the conditions in Theorem 2 are satisfied, we have (1) if ∆ 2 is bounded, then the proposed rule is asymptotically optimal and R P ROP (T ) (2) if ∆ 2 → ∞, then the proposed rule is asymptotically sub-optimal; (3) if ∆ 2 → ∞ and ξ n ∆ 2 2 → 0, then the proposed rule is asymptotically optimal.
Theorem 2 provides the convergence rate of the proposed classification rule for the twoclass problem with respect to ξ n . Base on such a result, Theorem 3 demonstrates that the property of the proposed classification rule (optimality or sub-optimality) depends on the scenarios of the true model's ∆ 2 . Specifically, (1) when ∆ 2 is bounded, i.e. lim n→∞ R Bayes > 0, then R P ROP (T ) converges in probability to the same limit as R Bayes .
R Bayes → 0, then R P ROP (T ) P → 0; in this case, if we further have ξ n ∆ 2 2 → 0, then R P ROP (T ) and R Bayes have the same convergence rate.
Next, we derive the consistency property for the proposed estimate of y. Denote byŷ P the predicted value of y obtained from the proposed model. Defineŷ B to be the predicted value of y for x when all parameters are known. Specifically, first obtain the class label k via the Bayes Now we establish the theoretical results ofŷ P in Theorem 4. This result compares the MSE of the proposed estimate of y with that from the optimal Bayes rule (under all parameters known). Since the classification errors from a classification rule might be larger than 0, the MSE ofŷ may not converge to 0 even though the sample size n is sufficiently large. Here we adopt the MSE Bayes as a reasonable performance benchmark to evaluate the property of the proposed model with respect to y. Theorem 4 states that the difference of MSE between the proposed model and the Bayes method converges to 0 in probability.

Two-class Settings of the Qualitative Response
In this section, we evaluate the performance of the proposed GAQQ method for a binary response Z under different inverse covariance matrices C and mean differences δ 2 . The proposed GAQQ model is compared with several benchmark methods, denoted as GLDA, CL, and ENET, which use the predictor variables X to predict Z and y. The GLDA employs the LDA classification rule for Z using the generalized inverse of the sample covariance matrix of X when p > n. The CL method applies the LPD technique introduced by Cai and Liu (2012) to predict the response variable Z based on X. With their estimated class label of Z, the GLDA and CL predict y by Equation (11). The ENET method uses the elastic-net logistic model (Zou and Hastie, 2005) on predictor variables X to fit the qualitative response Z and hence predicts Z for the testing data. For the quantitative response y, the ENET separately fits two elastic-net linear regressions for two classes using training data and then predicts y in the testing data based on its estimated Z. The tuning parameters of the CL and ENET methods are chosen by cross-validation.
Regarding the inverse covariance matrix C, we consider the following five structures in the simulation, which are commonly used in the literature (Yuan and Lin, 2007;Kang and Deng, 2020).
• Model 3. C 3 is generated by randomly permuting rows and corresponding columns of the matrix C 2 . This matrix has more sparsity as the dimensionality increases. Models 2 and 3 are relatively dense matrices, and they also become more sparse when the dimensionality increases. All of these four matrices have sparse structures to some extent, while Model 5 is a general sparse matrix with no structure, which is similarly used in Bien and Tibshirani (2011).
For the mean difference δ 2 , we consider two different levels of sparsity. The µ 1 is the vector with all elements zeros. Then generate µ 2 such that (S1): 25% of the elements in The same procedure is employed to generate the testing data, which is used to evaluate the prediction performance of y and Z for different compared methods. We consider the root mean squared prediction error RMSPE = 1 n n i=1 (y i −ŷ i ) 2 to measure the prediction accuracy for the quantitative response y, whereŷ i represents the predicted value. The prediction performance of the qualitative response Z is measured by the misclassification error ME = 1 n n i=1 I(z i =ẑ i ), whereẑ i is the predicted value of z i and I(·) is an indicator function.
Tables 1 and 2 report the averaged MEs in percentage and averaged RMPSE, as well as their corresponding standard errors in parenthesis for each approach over 100 replications. It can be seen from Table 1 that the proposed method generally outperforms other approaches with respect to MEs. Such an advantage becomes more significant as the underlying models are more sparse. Specifically, in the scenario of S1 = 25% and p = 40, the proposed GAQQ method does not perform as well as others, since the underlying models in this scenario are the least sparse, especially for the dense models 2 and 3. In contrast, the proposed method produces relatively better comparison results in the scenario of S2 = 75% and p = 40, where the true mean difference is more sparse. Furthermore, this advantage of the proposed method is well evidenced in the scenario of p = 80, and even more notable when p = 200 with its substantially lower MEs than other methods.
From Table 2, we observe that the proposed method generally gives superior performance over other compared approaches for each scenario in predicting the quantitative response y.
The possible explanations are in two folds. First, the proposed GAQQ method provides an accurate classification of the qualitative response Z. Second, the proposed GAQQ has a proper estimation of C by the regularization that is used in the prediction of quantitative response y according to (11), resulting in an improvement of the prediction accuracy. It is also seen that the CL and GLDA methods are comparable in some cases, possibly because both of them use the generalized inverse of the sample covariance of X forΣ −1 X in the prediction of quantitative response y in (11). But the CL method is generally better since it has more accurate classification results than the GLDA in Table 1.

Multi-class Settings of the Qualitative Response
Now, we examine the performance of the proposed GAQQ method for multi-class settings of the qualitative response. We consider p = 200 and K = 4 classes of qualitative response Z with training sizes n 1 = n 2 = n 3 = n 4 = 30 for Models 1 -5 of inverse covariance matrix C. Let µ kj represent the jth entry of the mean value µ k . Generate µ kj = 0.5 * k + u kj for j = 2k − 1, 2k, 2k + 1, . . . , 2k + 6, otherwise µ kj = 0, where u kj is from Unif (−1, 1). The training data are generated from N(µ k , C −1 ), and the testing data follow the same generation procedure. We compare the proposed method with the GLDA, as well as the estimators where the latter two methods are designed for multi-class problems. We use the WT and CHWE models to first predict the class label Z for the testing data, and then the response y is estimated, by the multivariate normal property, asμ ky +Σ ′ The results of performance measures, ME and RMSPE are summarized in Table 3 based on 100 replications. One can see that the GAQQ method performs better than the GLDA as well as the WT method, and is comparable with the CHWE in terms of the MEs. Besides, the GAQQ method gives the best performance among the compared approaches with significantly lower values of RMSPE.

Case Studies
In this section, we apply the proposed GAQQ method to two real-data case studies. The first one is from the study of Heusler compounds in material science and the second one is from the study of molecular diagnostics of Ulcerative colitis and Crohn's disease. Although from different fields, both problems contain QQ responses with high-dimensional predictors, and the proposed GAQQ method appears to have much better performance in terms of prediction accuracy compared with other methods.
The case study on material sciences is regarding the Heusler compounds, which are a large family of intermetallics with more than 1000 known members. Many Heusler compounds have shown exotic properties, such as superconductivity and topological band structures, which have promising applications for quantum computing. Understanding the thermody- namic stability of Heusler compounds lays the foundation for exploiting the large chemical space to discover and design new functional Heusler materials (Liu et al., 2016). To determine the thermodynamic stability of Heusler compounds, there are two key metrics: the mixing enthalpy (quantitative response) and the global stability based on hull energy (binary qualitative response). The comprehensive database of 180628 full Heusler structures was built by collecting the relevant structural and energetic data from the Materials Project (Jain et al., 2013), OQMD (Saal et al., 2013), and AFLOW (Curtarolo et al., 2012). These data were calculated using first-principles methods based on density functional theory, and it was extremely computationally expensive (taking hours) to generate one entry of the data.
Therefore, a statistical model that can accurately predict the thermodynamic stability for any elemental and compound features is a useful surrogate of the first-principle computation models.
Since there is an intrinsic relationship between two QQ responses, the proposed GAQQ method is suitable to improve the prediction accuracy by jointly fitting them together. To demonstrate the GAQQ method in the scenario when the number of predictors is large relative to the size of the data, we randomly choose 150 samples from each class of the binary response. We delete the predictor variables whose standard deviations are less than 1.0e −6 , resulting in 157 predictors of elemental and compound features. To examine the prediction performance of the GAQQ method and other comparison methods, we randomly divide data into a training set with a size of 200 and a testing set with a size of 100. Table   4 reports the prediction performance results based on 50 random splits of the Heusler data.
From the results, it is seen that the proposed GAQQ performs much better than other methods in comparison, with the smallest values for the misclassification error (ME) and the root mean squared prediction error (RMSPE). The second data for the case study considers the multi-class settings of the qualitative response. The IBD gene data (Burczynski et al., 2006)

Discussion
In this work, we propose a generative modeling approach to jointly model the data with QQ responses, which is a new perspective different from existing methods in the literature.
By fully exploring the joint distribution of the QQ responses and predictor variables, the proposed method enables efficient parameter estimation, accurate prediction, and lays a good foundation for investigating the asymptotic properties. The proposed model can be naturally extended to the situation for multiple quantitative responses.
One further research direction is to accommodate a more flexible structure on the joint distribution of QQ responses and predictor variables. For example, one can extend the LDA for the classification of the qualitative response to the quadratic discriminant analysis (QDA). The QDA is more flexible with different covariance structures in each class, but its estimation for high-dimensional data would encounter more difficulty due to a large number of parameters. Besides, the derivation of its asymptotic properties is much more technically complicated (Li and Shao, 2015). Another research direction is to apply the generative modeling approach for the data with semi-continuous responses (Wang et al., 2020), or the ordinal and quantitative responses. One may employ the ordinal regression for the ordinal response, and then derive its joint likelihood function with appropriate regularization.

Appendix
Derivation from (9) to (10). Let C denote a generic constant thereafter.
Derivation from (16) to (17). For δ j , j = 2, 3, . . . , K, where Hence, formula (20) is equal to , where x and y are multivariate variables. For a given value of Proof. We need to search for y to minimize where C is a constant not depending on y. Taking derivative of L(y) and setting to zero yields ∂L(y) ∂y This, together with a property of block matrix that Ω ′ XY = −Ω Y Σ ′ XY Σ −1 X , completes the proof.
The inequality p(x ∈ G 1 |X = x) > p(x ∈ G 2 |X = x) in Proposition 2 indicates that the LDA rule assigns x to G 1 . Therefore, Proposition 2 implies that Step 2b of Algorithm 2 is equivalent to applying the LDA classification rule directly on x instead of w = (x ′ ,ŷ) ′ .
This fact enables us to give theoretical proof for the consistency properties of the proposed classification rule based on variable X rather than W = (X ′ , y) ′ . Before the proof of Theorem 1, we present Lemmas 2 -4.
Write µ 0 k = ((µ 0 kX ) ′ , µ 0 ky ) ′ , where µ 0 kX is the true mean value of variable X for class G k . Correspondingly, writeμ k = (μ ′ kX ,μ ky ) ′ . Let a n ≍ b n represent two sequences a n and b n to be the same order. Now we state Lemma 3.
Proof. It is not difficult to derive Consequently, applying the Cauchy-Schwarz inequality together with Equations (25) and (26), we can obtain Similarly, As a result, according to Lemma 3, we have From Lemma 4, note that when t = k, we have (µ 0 With Lemmas 2 -4, we are ready to complete the proof of Theorem 1.
LetẐ P ROP andẐ Bayes denote the predicted class labels obtained by the proposed model and the Bayes rule, respectively. For simplicity, we assume π 1 = π 2 = . . . = π K instead of condition (C7) in the proofs of Theorems 1 -3 with no influence on the theoretical results, since condition (C7) is only used to bound the term log π k π 1 of the LDA rule. Define XδkX for a new sample x. Then for any ǫ > 0, ≤ Pr(|θ k − ϑ k | ≥ ǫ 2 for some k) + Pr(|ϑ k − ϑ l | ≤ ǫ for some k, l).
where the last inequality is obtained by condition (C6). Secondly, we bound the term Pr(|θ k −ϑ k | ≥ ǫ 2 for some k).
The conditional misclassification rate is By the result (18), we havê From the result (19), together with E[(δ 0 Hence we havê Therefore, by Lemma 3, we obtain Similarly, we haveδ which proves theory. To establish the theoretical results in Theorem 3, we need a lemma from Shao et al. (2011). We state it here for completeness, and then prove Theorem 3.

Lemma 5. Let a
(1) n and a (2) n be two sequences of positive numbers such that a (1) n → ∞ and a Proof. See the proof of Lemma 1 in Shao et al. (2011).


Generative Adversarial Transformers

We introduce the GANformer, a novel and efficient type of transformer, and explore it for the task of visual generative modeling. The network employs a bipartite structure that enables long-range interactions across the image, while maintaining computation of linear efficiency, that can readily scale to high-resolution synthesis. It iteratively propagates information from a set of latent variables to the evolving visual features and vice versa, to support the refinement of each in light of the other and encourage the emergence of compositional representations of objects and scenes. In contrast to the classic transformer architecture, it utilizes multiplicative integration that allows flexible region-based modulation, and can thus be seen as a generalization of the successful StyleGAN network. We demonstrate the model's strength and robustness through a careful evaluation over a range of datasets, from simulated multi-object environments to rich real-world indoor and outdoor scenes, showing it achieves state-of-the-art results in terms of image quality and diversity, while enjoying fast learning and better data-efficiency. Further qualitative and quantitative experiments offer us an insight into the model's inner workings, revealing improved interpretability and stronger disentanglement, and illustrating the benefits and efficacy of our approach. An implementation of the model is available at https://github.com/dorarad/gansformer.

Introduction
The cognitive science literature speaks of two reciprocal mechanisms that underlie human perception: the bottom-up processing, proceeding from the retina up to the cortex, as local elements and salient stimuli hierarchically group together to form the whole [27], and the top-down processing, where surrounding global context, selective attention and prior knowledge inform the interpretation of the particular [32]. While their respective roles and dynamics are being  actively studied, researchers agree that it is the interplay between these two complementary processes that enables the formation of our rich internal representations, allowing us to perceive the world around in its fullest and create vivid imageries in our mind's eye [13,17,39,52].
Nevertheless, the very mainstay and foundation of computer vision over the last decade -the Convolutional Neural Network, surprisingly, does not reflect this bidirectional nature that so characterizes the human visual system, and rather displays a one-way feed-forward progression from raw sensory signals to higher representations. Unfortunately, the local receptive field and rigid computation of CNNs reduce their ability to model long-range dependencies or develop holistic understanding of global shapes and structures that goes beyond the brittle reliance on texture [26], and in the generative domain especially, they are linked to considerable optimization and stability issues [70] due to their fundamental difficulty in coordinating between fine details across the generated scene. These concerns, along with the inevitable comparison to cognitive visual processes, beg the question of whether convolution alone provides a complete solution, or some key ingredients are still missing. § I wish to thank Christopher D. Manning for the fruitful discussions and constructive feedback in developing the bipartite transformer, especially when explored within the language representation area, as well as for the kind financial support that allowed this work to happen. arXiv:2103.01209v4 [cs.CV] 29 Mar 2022 Figure 2. Bipartite Attention. We introduce the GANformer network, that leverages a bipartite structure to support long-range interactions while evading the quadratic complexity standard transformers suffer from. We present two novel attention operations over the bipartite graph: simplex and duplex, the former permits communication in one direction, in the generative context -from the latents to the image features, while the latter enables both top-down and bottom-up connections between these two dual representations.
Meanwhile, the NLP community has witnessed a major revolution with the advent of the Transformer network [65], a highly-adaptive architecture centered around relational attention and dynamic interaction. In response, several attempts have been made to integrate the transformer into computer vision models, but so far they have met only limited success due to scalabillity limitations stemming from its quadratic mode of operation.
Motivated to address these shortcomings and unlock the full potential of this promising network for the field of computer vision, we introduce the Generative Adversarial Transformer, or GANformer for short, a simple yet effective generalization of the vanilla transformer, explored here for the task of visual synthesis. The model utilizes a bipartite structure for computing soft attention, that iteratively aggregates and disseminates information between the generated image features and a compact set of latent variables that functions as a bottleneck, to enable bidirectional interaction between these dual representations. This design achieves a favorable balance, being capable of flexibly modeling global phenomena and long-range interactions on the one hand, while featuring an efficient setup that still scales linearly with the input size on the other. As such, the GANformer can sidestep the computational costs and applicability constraints incurred by prior works, caused by the dense and potentially excessive pairwise connectivity of the standard transformer [5,70], and successfully advance the generative modeling of compositional images and scenes.
We study the model's quantitative and qualitative behavior through a series of experiments, where it achieves stateof-the-art performance for a wide selection of datasets, of both simulated as well as real-world kinds, obtaining particularly impressive gains in generating highly-structured multi-object scenes. As indicated by our analysis, the GANformer requires less training steps and fewer samples than competing approaches to successfully synthesize images of high quality and diversity. Further evaluation provides ro-bust evidence for the network's enhanced transparency and compositionality, while ablation studies empirically validate the value and effectiveness of our approach. We then present visualizations of the model's produced attention maps, to shed more light upon its internal representations and synthesis process. All in all, as we will see through the rest of the paper, by bringing the renowned GANs and Transformer architectures together under one roof, we can integrate their complementary strengths, to create a strong, compositional and efficient network for visual generative modeling.

Related Work
Generative Adversarial Networks (GANs) [28], originally introduced in 2014, have made remarkable progress over the past years, with significant advances in training stability and dramatic improvements in image quality and diversity. that turned them to be nowadays one of the leading paradigms in visual synthesis [5,44,58]. In turn, GANs have been widely adopted for a rich variety of tasks, including imageto-image translation [40,72], super-resolution [47], style transfer [12], and representation learning [18], to name a few. But while generated images for faces, single objects or natural scenery have reached astonishing fidelity, becoming nearly indistinguishable from real samples, the unconditional synthesis of more structured or compositional scenes is still lagging behind, suffering from inferior coherence, reduced geometric consistency and, at times, a lack of global coordination [9,43,70]. As of now, faithful generation of structured scenes is thus yet to be reached.
Concurrently, the last years saw impressive progress in the field of NLP, driven by the innovative architecture called Transformer [65], which has attained substantial gains within the language domain and consequently sparked considerable interest across the deep learning community [16,65]. In response, several attempts have been made to incorporate self-attention constructions into vision models, most commonly for image recognition, but also in segmentation [25], detection [8], and synthesis [70]. From structural perspective, these can be roughly divided into two streams: those that apply local attention operations, failing to capture global interactions [14,37,56,57,71], and others that borrow the original transformer structure as-is and perform attention globally across the entire image, resulting in prohibitive computation due to the quadratic complexity, which fundamentally hinders its applicability to low-resolution layers only [3,5,19,24,41,66,70]. Few other works proposed sparse, discrete or approximated variations of self-attention, either within the adversarial or autoregressive contexts, but they still fall short of reducing memory footprint and computational costs to a sufficient degree [11,24,36,38,62].
Compared to these prior works, the GANformer stands out as it manages to avoid the high costs ensued by self attention, employing instead bipartite attention between the image features and a small collection of latent variables. Its design fits naturally with the generative objective of transforming source latents into an output image, facilitating long-range interaction without sacrificing computational efficiency. Rather, the network maintains a scalable linear computation across all layers, realizing the transformer's full potential. In doing so, we seek to take a step forward in tackling the challenging task of scene generation. Intuitively, and as is later corroborated by our findings, allocating multiple latents to interact through attention with the generated image serves as a structural prior of a bottleneck that promotes the formation of compact and compositional scene representations, as the different latents may specialize to certain objects or semantic regions of interest. Indeed, as demonstrated in section 4, the Generative Adversarial Transformer achieves state-of-the-art performance in synthesizing varied real-world indoor and outdoor scenes, while showing indications for semantic disentanglement along the way. In designing our model, we draw inspiration from multiple lines of research on generative modeling, compositionality and scene understanding, including techniques for scene decomposition, object discovery and representation learning. Several variational approaches [7,22,23,31] perform iterative inference to encode scenes into multiple slots, but are mostly applied in the contexts of synthetic and oftentimes fairly rudimentary 2D settings. Works such as Capsule networks [29,61] leverage ideas from psychology about Gestalt principles [34,63], perceptual grouping [6] or analysis-bysynthesis [4], and like us, introduce ways to piece together visual elements to discover compound entities and, in the cases of Set Transformers [48] or A 2 -Nets [10], group local information into global aggregators, which proves useful for a broad spectrum of tasks, spanning unsupervised segmentation [30,50], clustering [48], image recognition [2], NLP [59] and viewpoint generalization [46]. However, our work stands out incorporating new ways to integrate information across the network through novel forms of attention: (Simplex and Duplex), that iteratively update and refine the assignments between image features and latents, and is the first to explore these techniques in the context of high-resolution generative modeling.
Most related to our work are certain GAN models for conditional and unconditional visual synthesis: A few methods [21,33,54,64] utilize multiple replicas of a generator to produce a set of image layers, that are then combined through alpha-composition. As a result, these models make quite strong assumptions about the independence between the components depicted by each layer. In contrast, our model generates one unified image through a cooperative process, coordinating between the different latents through the use of soft attention. Other works, such as SPADE [55,73], employ region-based feature modulation for the task of layout-to-image translation, but, contrary to us, use fixed segmentation maps and static class embeddings to control the visual features. Of particular relevance is the prominent StyleGAN model [44,45], which utilizes a single global style vector to consistently modulate the features of each layer. The GANformer generalizes this design, as multiple style vectors impact different regions in the image concurrently, allowing for spatially finer control over the generation process. Finally, while StyleGAN broadcasts information in one direction from the single global latent to the local image features, our model propagates information both from latents to features and vice versa, enabling topdown and bottom-up reasoning to occur simultaneously 1 .

The Generative Adversarial Transformer
The Generative Adversarial Transformer (GANformer) is a type of Generative Adversarial Network, which involves a generator network (G) that maps random samples from the latent space to the output space (e.g. an image), and a discriminator network (D) which seeks to discern between real and fake samples [28]. The two networks compete with each other through a minimax game until reaching an equilibrium. Typically, each of these networks consists of multiple layers of convolution, but in the GANformer case, we instead construct them using a novel architecture, called Bipartite Transformer, formally defined below.
The section is structured as follows: we first present a formulation of the Bipartite Transformer, a domain-agnostic generalization of the Transformer 2 (section 3.1). Then, we provide an overview of how the transformer is incorporated into the generative adversarial framework (section 3.2). We conclude by discussing the merits and distinctive properties of the GANformer, that set it apart from the traditional GAN and transformer networks (section 3.3).

The Bipartite Transformer
The standard transformer network is composed of alternating multi-head self-attention and feed-forward layers. We refer to each pair of self-attention and feed-forward operations as a transformer layer, such that a transformer is considered to be a stack of several such layers. The Self-Attention layer considers all pairwise relations among the input elements, updating each one by attending to all the others. The Bipartite Transformer generalizes this formulation, featuring instead a bipartite graph between two groups of variables -in the GAN case, latents and image features. In as a biologically-accurate reflection of cognitive top-down processing. Rather, this analogy plays as a conceptual source of inspiration that aided us through the idea development. 2 By transformer, we precisely mean a multi-layer bidirectional transformer encoder, as described in [16], which interleaves selfattention and feed-forward layers. the following, we consider two forms of attention that could be computed over the bipartite graph -Simplex attention and Duplex attention, depending on the direction in which information propagates 3 -either in one way only, from the latents to the image, or both in top-down and bottom-up ways. While for clarity purposes, we present the technique here in its one-head version, in practice we make use of a multi-head variant, in accordance with prior work [65].

SIMPLEX ATTENTION
We begin by introducing the simplex attention, which distributes information in a single direction over the bipartite transformer graph. Formally, let X n×d denote an input set of n vectors of dimension d (where, for the image case, n = W ×H), and Y m×d denote a set of m aggregator variables (the latents, in the generative case). We can then compute attention over the derived bipartite graph between these two groups of elements. Specifically, we define: Where q(·), k(·), v(·) are functions that respectively map elements into queries, keys, and values, all maintaining dimensionality d. We also provide the mappings with positional encodings, to reflect the distinct spatial position of each element e.g. in the image (see section 3.2 for details). Note that this bipratite attention is a generalization of self attention, where Y = X.
We can then integrate the attended information with the input elements X, but whereas the standard transformer implements an additive update rule of the form: we instead use the retrieved information to control both the scale as well as the bias of the elements in X, in line with the practice promoted by the StyleGAN model [44]. As our experiments indicate, such multiplicative integration enables significant gains in the model's performance. Formally: Where γ(·), β(·) are mappings that compute multiplicative and additive factors (scale and bias), both maintaining a dimension of d, and ω(X) = X−µ(X) normalizes the features of X 4 . By normalizing X (the image features), and then letting Y (the latents) control X's statistical tendencies, we essentially enable information propagation from Y to X, intuitively, allowing the latents to control the visual generation of spatial attended regions within the image, so as to guide the synthesis of objects and entities.

DUPLEX ATTENTION
We can go further and consider the variables Y to posses a key-value structure of their own [53]: where the values V store the content of the Y variables as before (i.e. the randomly sampled latent vectors) while the keys K track the centroids of the attention-based assignment between X and Y , which can be computed by K = a(Y, X) -namely, the weighted averages over X elements, using the attention distribution derived by comparing them to Y elements. Intuitively, each centroid tracks the region in the image X that interacts with the respective latent in Y . Consequently, we can define a new update rule: This update compounds together two attention operations: first (1) computing attention assignments between X and Y , by K = a(Y, X), and then (2) refining the soft assignments by considering their centroids, through A(Q, K, V ), where Q = q(X), which computes attention between the elements X and their centoroids K. This is analogous to the Expectation-Maximization or k-means algorithms, [49,50], where we iteratively refine the assignments of elements X to clusters Y based on their distance to their respective centroids K = a(Y, X). As is empirically shown later, this works more effectively than the update u s defined above.
Finally, to support bidirectional interaction between X and Y (the image and the latents), we can chain two reciprocal Figure 6. Sample images and attention maps of lower and upper GANformer layers, for the CLEVR, LSUN-Bedrooms, FFHQ and Cityscapes datasets. The colors in the attention maps correspond to the assignment between the image regions and the latent variables that control them. For the CLEVR dataset, we can see multiple attention maps produced by different layers of the model, revealing how the role of the latent variables changes at different stages of the generation -while they correspond to an instance segmentation as the layout of the scene is being formed in the early low-resolution layers, they behave similarly to a surface normal in the upper high-resolution layers of the generator. We see similar progression from a coarser to finer pattern of attention for the FFHQ dataset.

The Generator and Discriminator Networks
Our generator and discriminator networks follow the general design of prior work [44,45], with the key difference of incorporating the novel bipartite attention layers instead of the single-latent modulation that characterizes earlier models: Commonly, a generator network consists of a multilayer CNN that receives a randomly sampled vector z and transforms it into an image. The popular StyleGAN approach departs from this design and, instead, introduces a feed-forward mapping network that outputs an intermediate vector w, which in turn interacts directly with each convolution through the synthesis network, globally modulating the feature maps' statistics at every layer.
Effectively, this approach attains layer-wise decomposition of visual properties, allowing StyleGAN to control global aspects of the picture such as the pose, lighting conditions or color scheme, in a coherent manner over the entire image. But while StyleGAN successfully disentangles global attributes, it is more limited in its ability to perform spatial decomposition, as it provides no direct means to control the style of localized regions within the generated image.
The bipartite transformer offers a solution to accomplish this objective. Instead of modulating the style of all features globally, we use instead our new attention layer to perform adaptive region-wise modulation. As shown in figure 3 (right), we split the latent vector z into k components, z = [z 1 , ...., z k ] and, as in StyleGAN, pass each of them through a shared mapping network, obtaining a corresponding set of intermediate latent variables Y = [y 1 , ..., y k ]. Then, during synthesis, after each CNN layer of the generator, we let the feature map X and latents Y play the roles of the two element groups, mediating their interaction through our new attention layer -either simplex or duplex. This setting thus allows for a flexible and dynamic style modulation at the level of the region. Since soft attention tends to group elements based on their proximity and content similarity, we see how the transformer architecture naturally fits into the generative task and proves useful in the visual domain, allowing the model to exercise finer control in modulating local semantic regions. As we see in section 4, this capability turns out to be especially useful in modeling highly-structured scenes.
As to the loss function, optimization and training configurations, we adopt the settings and techniques used by Style-GAN2 [45], including in particular style mixing, stochastic variation, exponential moving average for weights, and a non-saturating logistic loss with lazy R1 regularization 5 . 5 In the prior version of the paper and in earlier stages of the model development, we explored incorporating the bipartite attention to both the generator and the discriminator, in order to allow both components make use of long-range interactions. However, in ablation experiments we observed that applying attention to the

Summary
To recapitulate the discussion above, the GANformer successfully unifies the GAN and Transformer architectures for the task of scene generation. Compared to traditional GANs and transformers, it introduces multiple key innovations: • Compositional Latent Space with multiple variables that coordinate through attention to produce the image cooperatively, in a manner that matches the inherent compositionality of natural scenes.
• Bipartite Structure that balances between expressiveness and efficiency, modeling long-range dependencies while maintaining linear computational costs.
• Bidirectional Interaction between the latents and the visual features, which allows the refinement and interpretation of each in light of the other.
• Multiplicative Integration rule to impact the features' visual style more flexibly, akin to StyleGAN but in contrast to the classic transformer network.
As we see in the following section, the combination of these design choices yields a strong architecture that demonstrates high efficiency, improved latent space disentanglement, and enhanced transparency of the generative process.

Experiments
We investigate the GANformer through a suite of experiments that study its quantitative performance and qualitative behavior. As we will see below, the GANformer achieves state-of-the-art results, successfully producing high-quality images for a varied assortment of datasets: FFHQ for human faces [44], the CLEVR dataset for multi-object scenes [42], and the LSUN-Bedrooms [69] and Cityscapes [15] datasets for challenging indoor and outdoor scenes. Notably, it even attains state-of-the-art FID scores for the challenging and highly-structured COCO dataset.
Further analysis we conduct in sections 4.1, 4.2 and 4.3 provides evidence for multiple favorable properties the GANformer posses, including better data-efficiency, enhanced transparency, and stronger disentanglement than prior approaches. Section 4.4 then quantitatively assesses the network's semantic coverage of the natural image distribution for the CLEVR dataset, while ablation and variation studies at section 4.5 empirically validate the necessity of each of the model's design choices. Taken altogether, our evaluation offers solid evidence for the GANformer's effectiveness and efficacy in modeling compsitional images and scenes.
generator only allows for stronger results, and so we have updated the paper accordingly. We compare our network with several approaches, including both baselines and leading models for image synthesis: (1) A baseline GAN [28] that follows the typical convolutional architecture 6 ; (2) StyleGAN2 [45], where a single global latent interacts with the evolving image by modulating its global style; (3) SAGAN [70], which performs self attention across all feature pairs in low-resolution layers of the generator and the discriminator; and (4) k-GAN [64] that produces k separated images, which are then blended through alpha-composition.
To evaluate all models under comparable training conditions, model size, and optimization scheme, we implement them all within our public codebase, which extends the official StyleGAN repository. All models have been trained with images of 256 × 256 resolution and for the same number of training steps, roughly spanning a week on 2 NVIDIA V100 GPUs per model (or equivalently 3-4 days using 4 GPUs). For the GANformer, we select k -the number of latent variables, from the range of 8-32. Note that increasing the value of k does not translate to an increased overall latent dimension, and we rather keep it equal across models. See section A for further implementation details, hyperparameter settings and training configurations.
As shown in table 1, our model matches or outperforms prior work, achieving substantial gains in terms of FID score, which correlates with image quality and diversity [35], as well as other commonly used metrics such as Precision and Recall (P&R) 7 . As could be expected, we obtain 6 In the baseline GAN, we input the noise through the network's stem instead of through weight modulation. 7 Note that while the StyleGAN paper [45] reports lower FID scores for FFHQ and LSUN-Bedrooms, they are obtained by training for 5-7 times longer than our experiments (specifically, they train for up to 17.5 million steps, producing 70M samples and demanding over 90 GPU-days). To comply with a reasonable compute budget, we equally reduced the training duration for all models in our evaluation, maintaining the same number of steps. the least gains for the FFHQ human faces dataset, where naturally there is relatively lower diversity in image layout. On the flip side, most notable are the significant improvements in performance for CLEVR, where our approach successfully lowers FID scores from 16.05 to 9.17, as well as LSUN-Bedrooms, where the GANformer nearly halves the FID score from 11.53 to 6.51, being trained for equal number of steps. These findings suggest that the GANformer is particularly adept at modeling scenes of high compositionality (CLEVR) or layout diversity (LSUN-Bedrooms). Comparing between the Simplex and Duplex Attentions further reveals the strong benefits of integrating the reciprocal bottom-up and top-down processes together.

Data and Learning Efficiency
We examine the learning curves of our and competing models (figure 7, (3)) and inspect samples of generated images at different stages of the training (figure 12). These results both indicate that our model learns significantly faster than competing approaches. In the case of CLEVR, it produces high-quality images in approximately 3-times less training steps than the second-best approach. To further explore the GANformer's learning aptitude, we perform experiments where we reduce the size of the dataset each model (and specifically, its discriminator) is exposed to during training to varying degrees ( figure 7, (4)). These results similarly validate the model's superior data-efficiency, especially where as few as 1k images are provided for training.

Transparency & Compositionality
To gain more insight into the model's internal representation and its underlying generative process, we visualize the attention distributions produced by the GANformer as it synthesizes new images. Recall that at each layer of the generator, it casts attention between the k latent variables and the evolving spatial features of the generated image.  As illustrated by figures 4 and 6, the latent variables tend to attend to coherent visual regions in terms of proximity and content similarity. Figure 6 provides additional attention maps computed by the model in various layers, showing how it behaves distinctively in different stages of the generation process. The visualizations imply that the latents carry a semantic sense, capturing objects, visual entities or other constituent components of the synthesized scenes. These findings can thereby attest to an enhanced compositionality that our model acquires through its multi-latent structure. Whereas prior work uses a single monolithic latent vector to account for the whole scene and modulate features at a global scale only, our design lets the GANformer exercise finer control that impacts features at the object granularity, while leveraging the use of attention to make its internal representations more structured and transparent.
To quantify the compositionality exhibited by the model, we use a pre-trained detector [67] to produce segmentations for a set of generated scenes, in order to measure the correlation between the attention cast by the latents with various semantic classes. Figure 8shows the classes that have the highest correlation with respect to the latent variables, indicating that different latents indeed coherently attend to semantic concepts such as windows, pillows, sidewalks or cars, as well as background regions like carpets, ceiling, and walls. This illustrates how the multiple latents are effectively used to semantically decompose the scene generation task.

Disentanglement
We consider the DCI and Modularity metrics commonly used in the disentanglement literature [20,60] to provide more evidence for the beneficial impact our architecture has on the model's internal representation. These metrics asses the Disentanglement, Completeness, Informativeness and Modularity of a given representation, essentially evaluating the degree to which there is a 1-to-1 correspondence between latent factors and global image attributes. To obtain the attributes, we consider the area size of each semantic class (e.g. cubes, spheres, floor), predicted by a pre-trained segmentor, and use them as the output response features for measuring the latent space disentanglement, computed over 1k images. We follow the protocol proposed by Wu et al. [68] and present the results in table 3. This analysis confirms that the GANformer's latent representations enjoy higher disentanglement compared to competing approaches.

Image Diversity
A major advantage of compositional representations is that they can support combinatorial generalization -a key foundation of human intelligence [1]. Inspired by this observation, we measure this property in the context of visual synthesis of multi-object scenes. We use a pre-trained object detector on generated CLEVR scenes, to extract the objects and properties within each sample. We then compute Chi-Square statistics on the sample set to determine the degree to which each model manages to cover the natural uniform distribution of CLEVR images. Table 2 summarizes the results, where we can see that our model obtains better scores across almost all the semantic properties of the scenes distribution. These metrics complement the common FID and PR scores as they emphasize structure over texture, or semantics over perceptual appearance, focusing on object existence, arrangement and local properties, and thereby substantiating further the model's compositionality.

Ablation Studies
To validate the usefulness of bipartite attention, we conduct ablation studies, where we vary the index of the earliest and latest layers of the generator network to which attention is incorporated. As indicated by figure 7 (1-2), the earlier (or lower resolution) attention begins being applied, the better the model's performance and the faster it learns. The same goes for the latest layer to apply attention to -as attention can especially contribute in high-resolutions, which benefit the most from long-range interactions. These studies provide a validation for the effectiveness of our approach in enhancing generative scene modeling.

Conclusion
We have introduced the GANformer, a novel and efficient bipartite transformer that combines top-down and bottom-up interactions, and explored it for the task of generative modeling, achieving strong quantitative and qualitative results that attest to the model robustness and efficacy. The GANformer fits within the general philosophy that aims to incorporate stronger inductive biases into neural networks to encourage desirable properties such as transparency, data-efficiency and compositionality -properties which are at the core of human intelligence, serving as the basis for our capacity to plan, reason, learn, and imagine. While our work focuses on visual synthesis, we note that the bipartite transformer is a general-purpose model, and expect it may be found useful for other tasks in both vision and language. Overall, we hope that our work will help progressing further in our collective search to bridge the gap between the intelligence of humans and machines.

Acknowledgments
We are grateful to Stanford HAI for the generous computational resources provided through Amazon AWS cloud credits. I also wish to thank Christopher D. Manning for the fruitful discussions and constructive feedback in developing the bipartite transformer, especially when we explored it for language representation, as well as for the kind financial support he provided that allowed this work to happen.
In the following, we provide additional experiments and visualizations for the GANformer model. First, we present in figures 12 and 9 a comparison of sample images produced by the GANformer and a set of baseline models, over the course of the training and after convergence respectively. Section A specifies the implementation details, optimization scheme and training configuration of the model. In section B and figure 8, we evaluate the spatial compositionality of the GANformer's attention mechanism, shedding light upon the roles of the different latent variables.

A. Implementation and Training Details
To evaluate all models under comparable conditions of training configuration, model size, and optimization details, we implement them all within the TensorFlow codebase introduced by the StyleGAN authors [44]. See table 4 for particular settings of the GANformer and table 5 for comparison of model sizes.
In terms of the loss function, optimization and training configuration, we adopt the settings and techniques used in the StyleGAN2 model [45], including in particular style mixing, Xavier Initialization, stochastic variation, exponential moving average for weights, and a non-saturating logistic loss with lazy a R1 regularization. We use Adam optimizer with batch size of 32 (4 × 8 using gradient accumulation), equalized learning rate of 0.001, β 1 = 0.0 and β 2 = 0.99 as well as leaky ReLU activations with α = 0.2, bilinear filtering in all up/downsampling layers and minibatch standard deviation layer at the end of the discriminator. The mapping layer of the generator consists of 8 layers, and ResNet connections are used throughout the model, for the mapping network, synthesis network and discriminator.
We train all models on images of 256 × 256 resolution, padded as necessary. The CLEVR dataset consists of 100k images, the FFHQ has 70k images, Cityscapes has overall about 25k images and LSUN-Bedrooms has 3M images. The images in the Cityscapes and FFHQ datasets are mirroraugmented to increase the effective training set size. All models have been trained for the same number of training steps, roughly spanning a week on 2 NVIDIA V100 GPUs per model.

B. Spatial Compositionality
To quantify the compositionality level exhibited by the model, we employ a pre-trained segmentor to produce semantic segmentations for the synthesized scenes, and use them to measure the correlation between the attention cast by the latent variables and the various semantic classes. We derive the correlation by computing the maxi- mum intersection-over-union between a class segment and the attention segments produced by the model in the different layers. The mean of these scores is then taken over a set of 1k images. Results presented in figure 8 for the LSUN-Bedrooms and Cityscapes datasets, showing semantic classes which have high correlation with the model attention, indicating it decomposes the image into semanticallymeaningful segments of objects and entities.  GAN StyleGAN2 k-GAN Figure 9. State-of-the-art comparison. A comparison between models' sample images for the CLEVR, LSUN-Bedrooms and Cityscapes datasets. All models have been trained for the same number of steps, which ranges between 5k to 15k kimg training samples. Note that the original StyleGAN2 model has been trained by its authors for up to 70k kimg samples, which is expected to take over 90 GPU-days for a single model. See next pages for comparison with further models. These images show that given the same training length the GANformer model's sample images enjoy higher quality and diversity compared to prior works, demonstrating the efficacy of our approach.
SAGAN VQGAN GANformer s Figure 10. A comparison of models' sample images for the CLEVR, LSUN-Bedrooms and Cityscapes datasets. See figure 9 for further description. Figure 11. A comparison between models' sample images for the CLEVR, LSUN-Bedrooms and Cityscapes datasets. See figure 9 for further description.

GAN
StyleGAN k-GAN Figure 12. State-of-the-art comparison over training. A comparison between models' sample images for the CLEVR, LSUN-Bedrooms and Cityscapes datasets, generated at different stages throughout the training. Sample images from different points in training are based on the same sampled latent vectors, thereby showing how the image evolves during the training. For CLEVR and Cityscapes, we present results after training to generate 100k, 200k, 500k, 1m, and 2m samples. For the Bedroom case, we present results after 500k, 1m, 2m, 5m and 10m generated samples during training. These results show how the GANformer, especially when using duplex attention, manages to learn a lot faster than competing approaches, generating impressive images early in the training.