extfieldsofstudy:
Materials Science
Physics
Computer Science

text:
Highly accurate local basis sets for large-scale DFT calculations in CONQUEST

Given the widespread use of density functional theory (DFT), there is an increasing need for the ability to model large systems (beyond 1,000 atoms). We present a brief overview of the large-scale DFT code Conquest, which is capable of modelling such large systems, and discuss approaches to the generation of consistent, well-converged pseudo-atomic basis sets which will allow such large scale calculations. We present tests of these basis sets for a variety of materials, comparing to fully converged plane wave results using the same pseudopotentials and grids.

Introduction
Over the last thirty years, density functional theory (DFT) has emerged as the leading electronic structure modelling technique, used in fields as diverse as biochemistry and electronic engineering, alongside physics, chemistry and materials science. The ability to model the atomic and electronic structure of molecules, liquids, nanoparticles and solids has become a key part of the scientific method.
However, almost all simulations are performed on a very restricted number of atoms: typically a few hundred, and very rarely beyond a thousand. There are various reasons for the restriction, the most often-cited of which is scaling: the asymptotic scaling with the number of atoms of standard DFT is cubic in time and quadratic in memory. While there has been considerable effort in developing linear scaling approaches to DFT [1,2] which have been demonstrated to be capable of calculations on millions of atoms [3,4], these introduce approximations (typically integrating over energy, so that eigenstates are not easily available, though can be calculated within a range [5]). There is therefore a significant gap in the sizes of systems that can be modelled exactly, lying in the range of thousands to tens of thousands of atoms.
There are numerous examples of simulations that require this size of system, for instance: realistic doping in semiconductors; biological molecules with or without explicit water; nanostructures; compounds with very dilute compositions; and large scale defects such as dislocations. There are, of course, many possible approaches to large scale electronic structure calculations. Approximate methods, such as density functional tight binding (DFTB) [6,7] are effective, but introduce some empiricism and/or fitting, reduce basis sets and remove explicit calculation of certain terms. Full DFT calculations can be made both more efficient and to scale better in parallel by judicious choice of basis sets. The largest example of a full DFT calculation used a real-space finite-difference approach to perform calculations on 10,000+ atom systems [8] with a demonstration that 100,000 atoms are possible [9].
Localised basis sets are a common choice for efficient, large-scale DFT approaches. Using such a basis, it is easy to form sparse matrices, and to diagonalise the Hamiltonian exactly. Examples of these approaches include the pseudo-atomic orbitals found in the widely-used siesta code [10] and the Gaussian basis sets in the CP2K code [11].
Our focus in this paper is the large-scale DFT code, Conquest, which is capable of exact calculations on systems with several thousand atoms [5,12] and linear scaling calculations on millions of atoms [3,4]. Conquest uses a basis set of support functions (SFs) to represent both the Hamiltonian and either the density matrix or the Kohn-Sham eigenstates. These support functions can be further represented by other functions: either a systematic basis of blip functions [13]; or numerical pseudo-atomic orbitals (PAOs) [14] (though Conquest often uses a 1:1 mapping of PAOs to SFs). Conquest can solve for the electronic ground state using three key methods: linear scaling DFT [3,15,16] which we will not discuss further here; multi-site support functions [17][18][19][20], where a small number of support functions are made from the PAOs from several atoms (hence multi-site), with the Hamiltonian solved by exact diagonalisation; and the primitive PAO basis set (one PAO is one SF), also solved using diagonalisation.
PAOs are convenient and easily generated, but are not systematic in their convergence. In this paper, we will describe our approach to generating basis sets of varying sizes, and will demonstrate that it is possible to produce relatively modest basis sets that closely reproduce converged plane-wave results using the same pseudopotentials.

Implementation details
The Conquest code has been described in detail elsewhere [15,16,[20][21][22]; nevertheless, some details of the implementation may be useful to assist the reader, and we summarise these here.
The central part of the Conquest code relates to the creation of matrices and their use in finding the ground state of the system being studied. When using PAOs as the basis set (as in this paper), we form the overlap matrix elements, and the kinetic and non-local pseudopotential contributions to the Hamiltonian matrix elements using the high accuracy approach pioneered by the siesta code [10]. The remaining parts of the Hamiltonian matrix elements (from the local part of the pseudopotential as well as the Hartree and exchange-correlation potentials) are calculated by integration on a uniform spatial grid.
The ground state of the system is found using either exact diagonalisation of the general eigenproblem (using Scalapack routines) or a linear scaling approach [15]. For exact diagonalisation, Bloch's theorem is applied and a Monkhorst-Pack mesh of points in reciprocal space is generated. For linear scaling, which is purely real-space, this is not needed. Forces [22] and stresses are calculated as exact derivatives of the energy (including Pulay forces and stresses from the movement of the basis functions).

Pseudopotentials
Over the last thirty years, pseudopotentials have become steadily more accurate; during this period, various approaches have been developed to soften and smooth the potentials, and reduce the necessary plane wave cutoff, including ultra-soft pseudopotentials (USPP) [23] and the projector-augmented wave (PAW) [24]. However, these add complications to any implementation, and it is important to note that Hamann's extension of norm-conserving pseudopotentials [25] to use multiple projectors for each angular momentum (as suggested by Vanderbilt [23]) permits extremely accurate normconserving pseudopotentials to be developed.
Another important recent development in the area of pseudopotentials is the "delta" study [26,27], which compared the accuracy of all the various pseudopotential methods to different all-electron approaches for a wide sampling of elemental solids. This study demonstrated that it is perfectly possible to develop libraries of norm-conserving pseudopotentials that are as accurate as the best PAW and USPP libraries. To date, there are two sets of these optimised norm-conserving (or ONCV) pseudopotentials [25] freely available: the PseudoDojo [28] and SG15 [29] libraries. We have developed a PAO generation code for Conquest which reads any ONCV pseudopotential generated by Hamann's code, which includes both of these sets. In this paper we use the potentials from the PseudoDojo library, regenerated for these tests using version 3.3.1 of Hamann's code, along with version 1.0.1 of the Conquest PAO generation code.
The only drawback to using these potentials is that almost all elements have partial core corrections [30] included, which can require a fine integration grid. Moreover, many of the heavier elements include semi-core states in the valence electrons, requiring more bands to be solved. These inclusions are necessary for high accuracy; we intend to develop interfaces to other pseudopotential generation schemes that will allow less accurate but faster calculations, which will be reported in future work.

Basis set sizes and defaults
All basis sets have advantages and disadvantages; the well-known advantages of plane waves (simplicity and systematic convergence) apply for relatively small systems, where the efficiency of fast Fourier transforms (FFTs) also applies; for large systems, however, the poor parallel scaling of FFTs becomes significant. Moreover, plane waves, being solutions for the free electron, are poorly adapted to atomic calculations (leading to the use of pseudopotentials) and more generally to the solid state (leading to large numbers of basis functions).
There are approaches to systematic basis sets with localised orbitals, where support functions (also known as non-orthogonal generalised Wannier functions, among other names) are defined by a radius and a grid spacing equivalent to a plane wave cutoff. Examples of these include: the blip functions used in Conquest [13]; periodic sinc functions [31,32]; wavelets [33]; and Lagrange functions [34]. It is also possible to use the grid points directly as basis functions with real space techniques [35] including finite elements [36,37] and finite differences [38].
There is, however, a simplicity and intuition which comes from using atomic orbitals, or (in the case of a pseudopotential calculation) pseudo-atomic orbitals, which consist of radial functions, normally tabulated on a fine radial mesh, multiplied by appropriate spherical harmonics. Integrals between basis functions can be simplified, with angular parts found analytically and radial grids so fine as to give effectively analytic results. The determination of the basis set then relies on three questions: how are the radial functions calculated?; how large an angular momentum is required?; and how many functions should there be for each angular momentum?
The question of angular momentum is in large part determined by the valence electrons being considered: these angular momenta must be represented in the basis set. It is then useful to consider polarisation functions; following the Siesta code [10] we can consider this as the effect of a local electric field on the highest valence shell. For elements up to the lanthanides, this ensures that the PAO basis set includes angular momenta up to l = 2; unless an atom contains valence f electrons, this is generally sufficient.
The radial functions are always found with some form of confinement (to ensure sparsity of matrices and efficiency). While this confinement can be a simple radial cutoff, it is not clear how to choose that cutoff: a smaller cutoff will give greater efficiency, but may not yield as good a basis function, particularly in materials where long-range or weak interactions are key. The Siesta code introduced the idea of a uniform energy shift for all pseudo-atomic orbitals [10] to allow for a consistent definition of confinement for different angular momenta. It also introduced a flexible confinement potential to allow optimisation of the radial functions [39].
Of course, with more than one radial function per angular momentum it is possible to choose different radii, but the question of how to set these is still difficult. The siesta code uses a split norm idea for subsequent radial functions: using a smooth polynomial from the origin to a certain point, and beyond that point matching the original. OpenMX calculates a set of five or six excited states (solutions with increasing numbers of nodes) for each angular momentum in a hard confining potential [40] and then combines these primitive functions into a number of composite functions (typically with two radial functions per valence state and one per polarisation state), fitted to dimers or diatomic molecules [41,42]. It is also possible to optimise confining potentials by fitting to the eigenstates of converged plane wave calculations [43]. As well as pseudoatomic orbitals, Gaussian orbitals are in use, for instance in CP2K [44]; Gaussian orbitals are the primitive basis for the original method on which Conquest's multi-site support functions are based [17].
In developing PAO basis sets for Conquest, we have considered the fact that the basis functions need to give variational flexibility to the Kohn-Sham eigenstates, and therefore need to span a range of radii. The simplest way, then, is to start with the idea of energy shifts and to specify radii on the basis of one energy that will give a highly confined orbital and one that will give a loosely confined orbital. We have found, empirically, that energy shifts of 2 eV and 0.02 eV fulfil these criteria well, and have also found that for situations where we specify a third radial function, using an intermediate value of 0.2 eV is effective. In fact, for most systems this gives three relatively evenly spaced confinement radii.
We define one set of PAOs by this equal energy criterion: a basis set with one radial function uses 0.02 eV; a set with two functions uses 2 eV and 0.02 eV; and with three functions uses 2 eV, 0.2 eV and 0.02 eV. We then define four basis sizes: minimal (one radial function, valence only or SZ); small (one radial function for valence and polarisation, or SZP); medium (two radial functions for valence and one for polarisation, or DZP); and large (three radial functions for all angular momenta, or TZTP) though in this paper we will not test the minimal basis.
We also consider an alternative, specifying equal radii for all angular momenta. We do this by finding the radii for all angular momenta for a given confinement energy, and taking their mean. This gives a uniform set of radii for all angular momenta. This option tends to be more efficient, as the overall radius of the largest support function is smaller than the equal energy construction; however, it can require a finer integration grid as the more diffuse functions are compressed. We will see below that the two approaches typically produce similar results, though the equal radii approach is maybe a little better overall.

Carbon
Silicon Germanium Basis

Tests
Our concern in this section is to test the accuracy of different PAO basis sets against the converged plane wave result. For this purpose we use the PWSCF code from the QuantumEspresso suite [45], which reads the same pseudopotentials as Conquest and allows a direct comparison. We note that we do not compare the results to experiments as this is not our aim: this type of comparison has been performed extensively elsewhere. We have chosen various solid systems for our tests: three of the elemental semiconductors (carbon, silicon and germanium, giving an excellent sampling of different gap sizes); simple oxides (two polymorphs of SiO 2 and MgO); perovskite oxides, including materials with semi-core states (SrTiO 3 , PbTiO 3 in the cubic phase and the distorted perovskite MgSiO 3 ); an example of a metallic system, non-magnetic bcc iron; and two weakly bonded systems (the ordinary form of ice, ice XI, which has hydrogen bonds; and a layered material, hexagonal boron nitride). These systems represent different forms of bonding and environment, and offer a good test of the different basis sets sizes and truncation approaches.
The calculations used either the PBE functional [46] (Fe, SiO 2 , MgO, MgSiO 3 , ice XI and BN) or the PBEsol functional [47] (C, Si, Ge, SrTiO 3 and PbTiO 3 ); the differences between the functionals are relatively small, though PBEsol is considered generally better for solids. The same functional has been used for all calculations of the same system (both QE and Conquest). We use the LibXC library [48] for PAO generation and Conquest calculations.
Conquest uses a regular grid for certain integrals, which is set using an energy cutoff. In these calculations the energy has been converged with respect to this grid. In plane wave calculations the charge density grid is generally taken to have an energy cutoff four times that of the plane waves, and we note that the Conquest grid energy cutoff is often around four times the QE plane wave cutoff. Details of cutoffs and Brillouin zone sampling (using Monkhorst-Pack grids) are given in the table captions. For the elemental semiconductors, shown in Table 1, we see good agreement with  Table 1.  Table 2. Parameters for SiO 2 in α-quartz and stishovite structures. Note that these are non-cubic, so the volume is given. SiO 2 calculations used a plane wave cutoff of 40Ha and an integration grid cutoff of 200Ha with reciprocal space meshes of 3 × 2 × 3 and 3 × 3 × 6 for α-quartz and stishovite, respectively. MgO used a plane wave cutoff of 60Ha, an integration grid of 260Ha and a reciprocal space mesh of 4 × 4 × 4.
DZP basis sets (differences up to 2-3% of the bulk modulus and 0.5% of the lattice constant) and excellent agreement with TZTP basis sets (differences typically less than 1% of bulk modulus and 0.2% of lattice constant). For context, when plotting binding energy curves (energy versus volume), a difference of 1% in the bulk modulus can barely be distinguished (after correcting for differences in lattice parameters). We illustrate the worst agreement, for Ge, in Fig. 1 and the PAOs generated with equal radii (note that all the curves are plotted relative to the fitted minimum volume and energy). In this case, the TZTP basis set gives very slightly worse agreement than the DZP, which is almost indistinguishable from the fully converged plane wave result. The SZP is a little too soft, but is still respectable. There is little difference between the equal radii and equal energy basis sets, except at SZP where the equal energy approach is maybe a little better. The simple oxides, with data found in Table 2, show similar performance to the elemental semiconductors. The TZTP basis sets are converged to within 1% of the bulk modulus for the equal radii, but to within 2% for the equal energies, while the  Table 3. Parameters for cubic perovskites SrTiO 3 and PbTiO 3 and the distorted perovskite MgSiO 3 . STO and PTO used a plane wave cutoff of 40Ha, an integration grid cutoff of 350Ha and a reciprocal space mesh of 9 × 9 × 9. MgSiO 3 used a plane wave cutoff of 50Ha, an integration grid cutoff of 200Ha and a reciprocal space mesh of 3 × 3 × 2.
equilibrium volumes are all with 1% (for both α-quartz and stishovite the unit cells are non-cubic, so we use equilibrium volume rather than lattice constant). A more challenging test for default PAOs (rather than PAOs tuned or optimised to an environment) is to compare the equilibrium phase (alpha quartz) to a high pressure phase (stishovite). The coordination of the Si atoms changes from four to six between these two phases. We compare the energy difference per formula unit (i.e. per SiO 2 unit) for the basis sets to the converged plane-wave result (-0.28 eV/unit). The TZTP basis sets give a good comparison (-0.20 eV/unit for equal energies and -0.22 eV/unit for equal radii). The DZP basis sets using perturbative polarisation (the default setting) give the correct ordering (-0.08 eV/unit for equal energies, and -0.05 eV/unit for equal radii) if a less accurate magnitude; however, generating l = 2 orbitals simply as excited eigenstates of the confined atom gives the incorrect ordering for both DZP (+0.15 eV/unit for equal energies and +0.19 eV/unit for equal radii) and SZP (+1.46 eV/unit for equal energies and +0.52 eV/unit for equal radii; these values are improved with perturbative polarisation but still have the incorrect sign). To test the dependence further, we added a second polarisation function (giving DZDP), which has an excellent result: -0.19 eV/unit for equal energies, and -0.20 eV/unit for equal radii (for perturbative polarisation functions; non-perturbative functions give a similar result). Clearly comparisons of stability of structures require radial flexibility in all angular momentum channels, and we would recommend use of at least DZDP whenever considering this kind of problem with default (i.e. not optimised) basis sets. For structural properties of the individual phases, however, the performance of the DZP basis sets is reasonable, though not quite as accurate as for the elemental semiconductors.
The perovskite structures, shown in Table 3, feature elements with semi-core states (4s and 4p states in Sr and 3s and 3p in Ti; the 5d states for Pb might be considered semicore but are not in this case) which are described with a single radial function. MgSiO 3 combines the elements seen in the simple oxides (we note that the pseudopotential in  Table 5. Ice XI relaxed cell parameters (using PBE). Calculations used a plane wave cutoff of 40Ha, an integration grid cutoff of 150Ha and a reciprocal space mesh of 3 × 2 × 2.
use here includes the 2s and 2p states, and that they are treated as semi-core). The performance is excellent for PbTiO 3 and good for SrTiO 3 (though the bulk modulus for TZTP with equal radii is surprisingly inaccurate) and the equilibrium volume for both these materials is very close to the plane wave result. It is possible that treating the Sr 4p states as valence, with further radial functions, might improve the performance. The bulk modulus for MgSiO 3 is significantly worse than either MgO or SiO 2 , with errors of nearly 5%. Nevertheless, these results give confidence in the default basis sets. Metallic bonding is very different to the covalent and ionic bonding studied thus far, but the performance of the default basis sets for non-magnetic bcc Fe, shown in Table 4, is excellent (we chose non-magnetic Fe simply for convenience; Conquest is capable of spin-polarised operation as simply as non-spin-polarised). In this case, both TZTP basis sets reproduce the plane wave results (the equal energy case has errors of 0.5% while the equal radii 2% in bulk modulus) while the DZP give excellent results. The equal energy SZP is still reasonable, though the equal radii is a little inaccurate. (For the Fe atom, both 3s and 3p states are included as semi-core states.) Weakly bound systems offer a larger challenge to local orbital basis sets (putting to one side the issues that DFT has with these systems, which will be the same for any other basis set). We start with hydrogen bonding, considering the optimum unit cell for  Table 6. BN optimum inter-layer distances and minimum energies for PBE and dispersion corrected (DFT-D2) PBE. Calculations used a plane wave cutoff of 50Ha, an integration grid cutoff of 150Ha and a reciprocal space mesh of 3×2×1. Conquest calculations used a counterpoise correction [49] ice XI, the ordinary form of ice, shown in Table 5. With a TZTP basis set, the three parameters are all accurate, comparing to the PW result to better than 1%, while DZP and SZP are much less accurate. This is the first case we have found where the use of a default (non-optimised) DZP basis set might cause a significant error. It is interesting to see how effective the TZTP basis sets are, even for a difficult system like this. Finally we turn to the layered material, boron nitride, which has dispersion interactions between layers, where the range of the PAOs may play a key role in the DFT part of the interaction; for the Conquest calculations, counterpoise corrections were used [49] to account for basis-set superposition errors. Data for the minimum distance between layers, and the resulting interaction energy is shown in Table 6. We report results for both standard PBE and PBE with semi-empirical dispersion corrections [50]; again, we note that we are testing the accuracy of our default PAO basis sets against converged PW calculations (and not the accuracy of DFT).
Good accuracy in comparison to the PW results is achieved with DZP and TZTP for the equal radii and equal energy basis sets for both approaches, though the equal energy DZP basis is perhaps a little high in the PBE interaction energy. The SZP basis set is less accurate, though it is interesting that the inter-layer spacing is rather close with the PBE-D2 approach, suggesting that the dispersion corrections mask the basis set errors.

Conclusions
We have introduced an approach to the construction of PAO basis sets for the Conquest code based on increasing energies (spaced by factors of ten). The radial functions are either based on a geometric increase in energy (equal energies) or on equal radii based on the average of the energies for all angular momenta (equal radii). These are both implemented in the Conquest PAO generation code which will be made available with the Conquest code on its release.
We have compared the performance of small (SZP), medium (DZP) and large (TZTP) basis sets with exact diagonalisation to converged plane wave results using the same functionals and pseudopotentials. Overall, the large (TZTP) basis sets are shown to reproduce the plane wave results to better than 1% in bulk modulus and often within 0.1% of the lattice constant for a variety of solids. Except for weakly bonded systems (particularly ice) we found that the medium (DZP) basis sets are nearly as accurate as the large basis sets and will typically be 5-10 times faster (simply from the reduction in matrix size). In general, both approaches (equal energies and equal radii) produce reliable results, though the equal radii approach gives slightly better results overall, and with smaller overall radii will give better efficiency. Alongside results from codes such as FHI-aims [51], these results demonstrate that local orbital codes can be as accurate as converged plane wave codes.


Predicting Sentiment of Polish Language Short Texts

The goal of this paper is to use all available Polish language data sets to seek the best possible performance in supervised sentiment analysis of short texts. We use text collections with labelled sentiment such as tweets, movie reviews and a sentiment treebank, in three comparison modes. In the first, we examine the performance of models trained and tested on the same text collection using standard cross-validation (in-domain). In the second we train models on all available data except the given test collection, which we use for testing (one vs rest cross-domain). In the third, we train a model on one data set and apply it to another one (one vs one cross-domain). We compare wide range of methods including machine learning on bag-of-words representation, bidirectional recurrent neural networks as well as the most recent pre-trained architectures ELMO and BERT. We formulate conclusions as to cross-domain and in-domain performance of each method. Unsurprisingly, BERT turned out to be a strong performer, especially in the cross-domain setting. What is surprising however, is solid performance of the relatively simple multinomial Naive Bayes classifier, which performed equally well as BERT on several data sets.

Introduction
Automated sentiment analysis usually involves training machine learning or deep learning models in supervised fashion. Typically, studies involve one data type and report high accuracy. For example, the seminal machine learning studies on IMDB movie reviews of (Pang et al., 2002) indicated accuracy over 80%. Recently, the accuracy of a deep learning system reported on this data set exceeded 95% (Howard and Ruder, 2018). Similarly, sentence-level sentiment predictions exceeded 95% accuracy on binary version of the popular Stanford Sentiment Treebank (abbreviated as SST-2) (Liu et al., 2019).
However, multiple studies indicate that the models trained on such data sets are in fact very far from being applicable universally. Machine and deep learning models tend to fit to specific type of texts and language. When applied to different language types in terms of both style and vocabulary, the performance drops sharply. This issue is often explored under the name of domain dependency or data set dependency. Several methods have been proposed to address it, such as for example (Selmer et al., 2013). In another stream of related studies, authors considered the task of cross-domain sentiment analysis: adaptation of a model to target domain or text type that is different from the domain or text type that the model was trained on. Examples include (Peng et al., 2018).
The goal of this article is to evaluate and compare supervised techniques of sentiment analysis of short texts using all available resources in the Polish language. We utilize three generations of machine learning: • machine learning models using bag-of-words vector representations, algorithms such as Naive Bayes and Support Vector Machines, • recurrent deep neural networks based on LSTM with and without pre-trained word embeddings, • finally the most recent generation of deep neural networks, pre-trained on language modeling tasks (BERT and ELMO).
The questions that our paper addresses are: (1) to what extent is each of the method usable for training a universal sentiment analysis model (how well it performs in cross-domain setting) and (2) how good it is for predicting sentiment within the same text type it has been trained on (how well it performs in in-domain setting).
The motivation for a universal classifier might not be obvious at first, as clearly the best performance is achieved by in-domain classification (Twitter may need tweet classifier, Facebook needs posts classifier, IMDB needs review classifier, and so on). However, universal character of sentiment classification models allows for better performance in cases when source data distribution is different than target data distribution. This happens on every day basis, for instance Twitter posts change their topic over time and optimum performance requires new train sets to match the newest data. These issues are well-known to machine learning community and explored under topics of domain adaptation, dataset shift and semantic drift. Ability to apply the model universally to various data types is a great advantage from the practical point of view.
Our intention is to focus on supervised learning. Specifically, this means working on collections of short texts such as single sentences, tweets or short reviews with labeled sentiments. The goal of our task is to classify sentiment of the whole written utterance (as for example, a sentence, review or tweet) into three classes: as positive, neutral or negative.
When computing sentiment we rely only only on learning from provided text examples and do not use any resource which might help in sentiment predictions such as a sentiment dictionary.
The paper is organized as follows: Section 2 contains a description of sentiment datasets, Section 3 describes the methods used for sentiment prediction. Section 4 describes the results obtained in our experiments. Finally, Section 5 contains conclusions and a discussion of possible future work.

Data Sets
The experiments described in this paper are based on several resources with manually labeled sentiment. The texts in our experiments are "short": tweets do not exceed 140 characters, Treebank sentences are arbitrarily long in terms of tokens but syntactically and semantically correct, reviews contain from 1 to 3 sentences.

Polish Sentiment Treebank (TW)
The first resource is Polish language dependency treebank with sentiment annotations ("Treebank Wydźwięku" abbreviated as TW). It is available to download 1 . Similar to Stanford's Sentiment Treebank (SST) (Socher et al., 2013), Treebank Wydźwięku (TW) was intended to study compositional phenomena in sentiment analysis. There are several notable differences between the two: • Dependency trees (TW) instead of constituency binary parse tress (SST), • 3-class sentiment (TW) instead of 5-class (SST), • Open-domain (TW) instead of one domain of movie reviews (SST).
In TW, overall sentiment of each sentence corresponds to the sentiment of its root. Sentences in this data set often contain mixed sentiment, even opposite polarities: one part may be positive and the other negative. This makes the task of predicting overall sentiment more difficult.

TW Version 1.0
Initial version of the TW treebank contained sentences from Składnica Treebank 2 (part referred to as sklad) and sentences from two types of product reviews: perfumes and clothes (part called rev).
The first release of the treebank was published as a part of Task 2: Sentiment analysis in Pol-Eval 2017 campaign on evaluation of natural language processing tools for Polish. In this competition, submitted tools competed against one another within certain tasks selected by organisers, using provided data. Solutions were evaluated according to common procedures.
The intended use of the treebank was as follows: given a set of syntactic dependency trees, the goal was to provide the correct sentiment for each subtree (phrase). Phrases correspond to sub-trees of dependency parse tree. Annotations assign sentiment values to whole phrases (and in some cases, sentences), regardless of their type. The PolEval part of the treebank and related evaluation script may be freely downloaded 3 .
Three systems participated in the tasks, all of them based on TreeLSTM (Tai et al., 2015). The description of systems and evaluation methodology can be found in (Wawer and Ogrodniczuk, 2017). Due to different nature of these tasks (computing sentiment of each sub-phrase and each sentence vs sentiment of sentences only) the results of these systems are not directly comparable to results reported in our paper.

TW Version 2.0
In August 2018, a new batch of sentences has been added to TW. It contains following new parts: • Test (evaluation) sentences from PolEval 2017 sentiment task (part called polevaltest), • 2 x 500 sentences collected from various web sources, mostly difficult, mixed sentiments and negative ones (parts called neg and jun18).
To our best knowledge, our paper is the first to describe and use the new version of this resource.

Twitter Data
This data set contains one thousand Polish language tweets, gathered and manually labeled as to their sentiment during the TrendMiner project 4 . Many tweets are related to publicly discussed events, some of them originate from politicians, journalists and public figures. Many of them are tweets of simple Twitter users, including teenagers. Overall, the dataset appears to be a fairly representative sample of communication occurring on Twitter in the Polish language. Due to Twitter's policy it can not be made publicly downloadable.

Movie Reviews
This data set consists of one thousand manually collected movie reviews from Polish website: http://www.filmweb.pl. Most of them are very short texts (1-2 sentences), the rest is a collection of reviews containing up to 5 sentences. Each review has a corresponding numeric score (number of stars) from 1 to 10, assigned by the review's author. Each category (number of stars) has exactly one hundred reviews. All scores were re-scaled into three categories: stars from 1 to 3 were re-scaled into negative category (-1), stars from 4 to 6 into neutral (0), stars 7 to 10 into positive (1).
What should be noted, however, is that the reviews very often contain spelling mistakes, words with omitted Polish diactric marks, often contain slang or sarcasm. All that makes them problematic for automated analysis.

Label Frequencies
Most of the datasets are not balanced in terms of sentiment label distributions. Twitter data set contains mostly neutral texts (tweets). Filmweb's balance is nearly perfect, as only the positive class has slight advantage in terms of frequency. Also TW's balance is far from perfect distribution between three sentiment classes as most sentences are neutral. Some pieces of TW treebank have been deliberately created to address imbalance issues in the sentiment treebank, such as for example part of the data called neg in TW 2.0, which contains many negative sentences. Table 1 presents sentiment label distribution in each of the data set. 3 Machine and Deep Learning Methods

Bag-of-Words and Machine Learning
Machine learning methods described in this subsection used bag-of-words text representations.
We converted text to word vectors using wordlevel unigram vectorizer with TF-IDF weights.
We have focused on two machine learning algorithms: Naive Bayes and Support Vector Machines.
Multinomial Naive Bayes (NB) Classifier is a well-known supervised machine-learning algorithm with an assumption of independence among predictors. Support Vector Machine (SVM) is also a wellknown supervised machine learning algorithm. We used linear kernels and implementation from the liblinear library 5 .

LSTM Neural Network (NN)
We used two approaches to implement the first layer of the neural network.
In the first approach we used untrained, random initialized embedding layer that uses 32 length vectors to represent each word. This method is marked as NN in the results.
In the second approach we changed this layer to pre-trained word2vec 100-dimensional word embeddings for Polish 6 . The embeddings were generated using gensim package (Řehůřek and Sojka, 2010) with skip-gram model architecture for two large text corpora: The National Corpus of Polish 7 and Polish edition of Wikipedia. We marked this approach as NN+E subsequently.
The structure of the neural network was as follows: word embedding layer, LSTM layer with 100 memory units, two Dense layers: the first with 100 neurons, the second with 3 output values, one for each class, with dropout regularization between them. As an output layer we used the softmax activation function.

ELMO
In one of the approaches we used ELMo method (Peters et al., 2018) to represent texts. ELMo, as it's authors put it, is a deep contextualized word representation that models both (1) characteristics of word use (e.g., syntax and semantics), and (2) how these uses vary across linguistic contexts (i.e., to model polysemy). These word vectors are extracted from states of a deep bidirectional language model, which is pre-trained on a large text corpus. We used the ELMO implementation and models described in (Che et al., 2018). The model was trained on Polish language Wikipedia. ELMO, although currently often superseded by other alternatives, contributed to state-of-the-art results in multiple natural language processing tasks such as question answering or paraphrase detection.
To obtain ELMO representation of each text, we computed average vector from 3 neural network layers, which resulted in a vector of 512 numbers. In the second step, these vectors were used to classify sentiment of an input text. Here, we experimented with multiple well-known machine learning methods such as Logistic Regression (LR), Random Forest (RF) with 200 trees and Support Vector Machine classifier (SVC) with a linear kernel. Each of these variants is subsequently marked as ELMO-LR, ELMO-RF and ELMO-SVC.

BERT
BERT is an example of the newest generation of pre-trained neural networks based on transformer architecture (Devlin et al., 2018). BERT acronym stands for Bidirectional Encoder Representations from Transformers, also obtained state-of-the-art results on multiple NLP tasks such as natural language inference and question answering. BERT model comes pre-trained on a very large corpus of unlabeled data, can be subsequently fine-tuned to a task with a limited amount of data such as sentiment analysis.
In our experiments we used smaller version of BERT. It contains 110M parameters and has support for 104 languages, 12 layers, the size of each hidden layer is 768, 12 self-attention heads (bertbase-multilingual-cased). We set maximum sequence length parameter to 128, which is enough to cover several sentences, and tested 3 and 4 training epochs. We tested BERT in a scenario which adds a sequence classification head on top. BERT Transformer is pre-trained, the sequence classification head is only initialized and has to be trained.

Results
This section illustrates the results of three experimental modes tested in our paper.

In-Domain
In the first mode (in-domain), we examine the performance of models trained and tested on the same text collection using standard cross-validation. On one hand, the performance of in-domain is driven up by lexical and structural similarity: training data are likely to be similar to test data in terms of vocabulary and syntactic structures. On the other,  The results of this mode are presented in Table 2. The best results were achieved by BERT (two treebank datasets: jun18 and neg) and SVC classifier (treebank dataset sklad, twitter and filmweb). Except for filmweb data set with over 30% discrepancy between the best and worst methods, the differences between methods were not large, usually did not exceed several percents.
On the whole TW sentiment treebank (marked as all-TW) the surprising winer was Naive Bayes, that managed to outperform other methods including BERT by a small margin.

One vs Rest Cross-Domain
In the second mode (one vs rest cross-domain) we train models on all available data except the given test collection, which we use as a test set. In this mode, models did not have a chance to learn from data similar to test data (maybe except some parts of TW treebank which may be considered similar). However, can utilize and possibly benefit from information contained in all other data sets available. Table 3 presents the results of one vs rest cross-domain experiments. In this mode, the best performers were BERT (the best results for 5 out 7 data sets) and surprisingly, Naive Bayes algorithm (best performance for 4 out of 7). Recurrent neural networks (NN) did not manage to reach state-of-the-art results, however it is easy to notice strong positive influence of pre-trained word2vec word embeddings (NN+E), with accuracy gains from several to as much as twenty percentage points in the case of twitter. One can also note that the performance of ELMO with Random Forest (ELMO+RF) is significantly better than the two other ELMO variants we tested.  In the third mode (one vs one cross-domain), we train a model on one data set and apply it to another one, repeating for each combination. In this mode we tested only the highest performing methods, such as BERT and Multinomial Naive Bayes. In this experiment we merged all sub-parts of the sentiment treebank (jun18, neg, polevaltest, rev, sklad) into one data set presented as TW. It consists of 2500 sentences. Table 4 contains one vs one results for BERT classifier. Rows refer to test data sets and columns to train data sets. As we can see, the accuracy of 85.4% on tweets with models trained on TW is high and identical to SVC of in-domain 5-fold cross-validation and also identical to BERT in one vs rest mode. The performance on filmweb indicates that the models did not start to learn effectively. Training on filmweb did not help the performance on TW treebank (only 15.3%). Since filmweb dataset is reasonable in size and balanced, we can only hypothesize that the language is too different. Table 5 contains one vs one results for Naive Bayes classifier. As before, rows refer to test data sets and columns to train data sets. In some cases Naive Bayes outperformed BERT. It was the case with training on filmweb movie reviews, models trained on this data set performed better than BERT both on tweets and on sentiment treebank TW. Also in the scenario of training on TW and testing on filmweb, Naive Bayes turned out to be better.

Conclusions and Future Work
The main point of this paper was to use all available Polish language data sets to seek the best possible performance in supervised sentiment analysis of short texts. We compared three generations of methods: machine learning with bag-of-words representation, recurrent neural networks (with and without pre-trained word embeddings) and finally deep neural networks pre-trained on language modeling task, including the newest transformer architecture BERT.
In sentiment classification, data available at training time is often different from data we intend to analyze in production environments. Ideally, classifiers should be capable of predicting sentiment on multiple types of data, covering various topics and texts of varied length without the need of re-training. In practice, achieving the best pos-sible performance requires training or re-training on data very similar to those we intend to analyze.
To explore the limits of this approach we experimented with cross-domain setting in which we train the model on one text type and apply it to another text type (one vs one cross-domain). We confirmed that this setting poses a problem often leading to substantial performance degradation.
Using several sentiment-labeled data sets as training data may in theory improve classifier's accuracy and robustness. In our paper we investigated possible benefits from training models on data less similar to the test set (cross-domain one-vsrest mode) and compared this to model training on smaller amounts of highly similar data (in-domain, models trained on the same type of data).
We found that for some data sets (TW treebank sub-sets, twitter) the results are comparable for cross-domain and in-domain setting, while for movie reviews in-domain setting turned out to be almost 30% better. Here, similar training data played a more significant role than using other less similar data sets to learn from. The bestperforming in-domain method turned out to be somewhat old SVC with simple bag-of-words representation.
BERT, transformer-based neural network with pre-training, turned out to benefit from large amounts of less similar data, with top performance in one-vs-rest cross-domain setting. Interestingly, multinomial Naive Bayes method turned out to perform on a very similar level with far less model parameters, which may be a viable alternative in more speed oriented environments without GPU processors.
Some of the issues raised in this paper are worth pursuing in further work. The first problem is the amount of pre-training and architecture changes needed to reach acceptable cross-domain performance. Apparently, this problem has still not been solved and more efforts are needed to reach high accuracy.
The second problem worth investigating is the matter of how suitable are sentiment treebanks, designed for experiments on within-sentence compositional sentiment phenomena (for example, studying sentiment propagation in sentence structure or mixed sentiments) for predicting single label of sentence-level sentiment. As a part of this future study, we intend to compare the performance of Tree-LSTM methods such as those re-


extfieldsofstudy:
Computer Science
Medicine
Mathematics

text:
HA-CCN: Hierarchical Attention-based Crowd Counting Network

Single image-based crowd counting has recently witnessed increased focus, but many leading methods are far from optimal, especially in highly congested scenes. In this paper, we present Hierarchical Attention-based Crowd Counting Network (HA-CCN) that employs attention mechanisms at various levels to selectively enhance the features of the network. The proposed method, which is based on the VGG16 network, consists of a spatial attention module (SAM) and a set of global attention modules (GAM). SAM enhances low-level features in the network by infusing spatial segmentation information, whereas the GAM focuses on enhancing channel-wise information in the higher level layers. The proposed method is a single-step training framework, simple to implement and achieves state-of-the-art results on different datasets. Furthermore, we extend the proposed counting network by introducing a novel set-up to adapt the network to different scenes and datasets via weak supervision using image-level labels. This new set up reduces the burden of acquiring labour intensive point-wise annotations for new datasets while improving the cross-dataset performance.

I. INTRODUCTION
T HE task of crowd counting is riddled with various challenges such as perspective distortion, extreme scale variations, heavy occlusion, illumination changes, clutter, nonuniform distribution of people, etc. Due to these issues, crowd counting and density estimation is a very difficult problem, especially in highly congested scenes. Several recent convolutional neural network (CNN) based methods for counting [1,2,3,4,5,6,7,8,9,10,11,12] have attempted to address one or more of these issues by adding more robustness to scale variations by proposing different techniques such as multicolumn networks [2], intelligent selection of regressors suited for a particular crowd scenario [3] and incorporating global, local context information into the counting network [4], etc. Methods such as [3,4,8] achieve significantly lower errors compared to the earlier approaches, however, they are complex to train due to the presence of multiple learning stages. For instance, Switching-CNN [3] involves different stages such as pre-training, differential training, switch training and coupled training. Similarly, CP-CNN [4] requires that their local and global estimators to be trained separately, followed by end-toend training of their density map estimator. Although the most recent methods such as [6,7,10] achieve better results while being efficient, there is still considerable room for further improvements. In this paper, we propose to improve the counting performance by explicitly modeling spatial pixel-wise attention and global attention into the counting network. Considering that crowd images have large variations in head sizes, it is essential to leverage multi-scale information by employing feature maps from different conv layers of the VGG16 network [13]. Several works such as [14,15,16,17] have demonstrated that different sized objects are captured by different layers in a deep network. Hence, an obvious approach would be to design a multi-scale counting network [18,19] that concatenates feature maps from different layers of the VGG16 network. However, earlier layers in a deep network capture primitive features and do not learn semantic awareness. Due to this, naive concatenation of feature maps from different layers of the network is not necessarily an optimal approach to address the issue of large scale variations in crowd images.
To address this issue, we introduce a spatial attention module (SAM) in the network, that is designed to infuse semantic awareness into the feature maps. This module takes the feature maps from lower layers as input, and learns to perform foreground-background segmentation. Furthermore, it uses this learned segmentation map to enhance the lower layer feature maps by selectively attending to specific spatial locations in this lower layer. Furthermore, we also attempt to augment channel-wise information in the higher level layers by employing a set of global attention modules. These modules selectively enhance important channels while suppressing the unnecessary ones. Fig. 1 provides an overview of the proposed attention-based feature concatenation for multi-scale crowd counting.
In addition to improving the count performance, another major issue in the crowd counting research community is the poor generalization performance of the existing networks. This is due to the fact that CNN-based methods are highly data-driven and suffer from inherent dataset bias. Hence, they cannot be applied directly to new scenes without further fine-tuning. A simple solution to this would be to train the model on the target dataset in a fully-supervised fashion, which requires expensive ground-truth annotations. Several earlier works such as [1,5] address this issue by proposing different semi-supervised or unsupervised fine-tuning methods in addition to their novel network designs. For instance, Zhang et al. [1] presented a cross-scene counting approach where they use perspective maps to retrieve candidate scenes from source dataset that are similar to the target set, which are then used to fine-tune the network. However, perspective maps may not be always available. Additionally, it is dependent on the assumption that their pre-trained model provides good estimates of count in the target patches. Liu et al. [5] proposed a self-supervised method based on image ranking to adapt to different datasets. While it achieves better generalization performance, their method is still limited since they use only unlabeled data. To address this generalization issue, we take a different approach as compared to earlier attempts ( [1,5]) by proposing a novel weakly supervised learning setup. We leverage imagelevel labels, which are much easier to obtain as compared to point-wise annotations 1 , in a weakly supervised fashion for fine-tuning networks to newer datasets/scenes. To achieve this weak supervision, we use the idea of image-level labeling of crowd images into different density levels by Sindagi et al. [4] and Fu et al. [20]. While these methods [4,20] employ image-level labels in conjunction to point-wise annotations to train their networks, we propose to use only image-level labels in the weakly supervised setup while adapting to new datasets, thereby avoiding the labour intensive point-wise annotation process. Fig. 2 illustrates the different types of annotations used for training the network. Fig. 2(a) represents samples from a source dataset, which consists of images and corresponding point-wise ground-truth annotations. The source dataset is used to pre-train the counting network. Fig. 2(b) represents samples from the target set to which we intend to adapt the pre-trained counting network. The pre-trained network is then fine-tuned on the target dataset using imagelevel labels via the proposed weakly supervised approach. During testing, the estimated density map from the fine-tuned network is compared with the the ground-truth using standard metrics as described later.
To summarize, the following are our key contributions in this paper: • A new network design that employs attention mechanisms at various levels for selectively enhancing the features from different layers of VGG16 network to increase the effectiveness of multi-scale concatenation. • A novel setup to adapt existing crowd counting models to new scenes and datasets via weakly supervised learning.
To the best of our knowledge, this is the first attempt to perform weak supervision using image-level labels for crowd counting.
In the following sections, we discuss related work (Section II) and the proposed method in detail (Section III). Details of experiments and results of the proposed method along with comparison on different datasets are provided in Section V, followed by conclusions in Section VI.
Babu et al. [8] proposed an automatically growing CNN to progressively increase the capacity of the network based on the dataset. Shen et al. [7] used adversarial loss similar to [4] to attenuate blurry effects in the estimated density maps. Shi et al. [6] proposed deep negative correlation based learning of more generalizable features. In another interesting approach, Liu et al. [5] proposed to leverage unlabeled data for counting by introducing a learning to rank framework. Li et al. [41] proposed CSR-Net, that consists of two components: a front end CNN-based feature extractor and a dilated CNN for the back-end. Ranjan et al. [10] proposed a network with two branches that estimates density map in a cascaded fashion. Cao et al. [9] proposed an encoder-decoder network with scale aggregation modules. They use a combination of Euclidean loss and a newly introduce local pattern consistency loss to train their network. Idrees et al. [42] proposed a new largescale crowd dataset with 1.25 million annotations, along with a novel loss function to train their dense-net [43] based architecture.
Recently, a few approaches have been proposed that incorporate detection of crowded regions through different techniques such as attention injective deformable network [44], semantic prior-based residual regression [45], use of auxiliary task such as segmentation [46] and segmentation infusion via inverse attention [47]. Other techniques such as [48] exploit multi-scale features in different ways. For instance, Jiang et al. [48,49,50] propose a Trellis style encoder decoder, where the multi-scale feature maps in the decoder are combined in an effective way. Similarly, Shi et al.and Liu et al. [49,50] exploit multi-scale features by explicitly considering perspective and context information respectively. Zhang et al. [51] address the issue of wide area counting by proposing a multi-view fusion CNN. Wang et al. [52] presented a new large-scale, diverse synthetic dataset and proposed a SSIM based CycleGAN [53] to adapt the synthetic datasets to real world dataset.
Attention mechanisms. Inspired by the role of attention in human visual perception [54,55,56], several works have successfully incorporated attention mechanism to improve the performance of CNNs for a variety of tasks such as image captioning [57,58], visual question answering [59,60,61,62], pose-estimation [63], classification [64,65,66], detection [67], fine-grained recognition [68], sequence to sequence modeling [69] etc. Xu et al. [62] were among the first to introduce visual attention in image captioning where they use different pooling mechanisms that attend to important and relevant regions in the scene. Zhu et al. [70] employed soft attention to combine image region features for the task of VQA. For the same task, Yang et al. [59] and Xu et al. [62] employed multiple stacked spatial attention models, in which the spatial attention map is successively refined. Chu et al.. [63] used attention to guide multi-contextual representation learning for improving pose estimation performance. Wang et al. [71] improved classification performance by incorporating 3D attention maps, generated using hour glass modules, into residual networks. Hu et al. [72] proposed a compact Squeeze-and-Excitation (SE) module to leverage inter-channel relationships. Recently, Woo et al. [64] extended the work of [66] by employing spatial and channel-wise attention modules after every layer in the network. In a different application, Zhu et al. [69] employed seq2seq in their decoder structure to model temporal video sequences. The attention mechanism at each step is used to help the decoder to decide which frames in the input sequence might be related to the next frame reconstruction. Note that they use attention mechanism to perform input selection. In contrast, we use attention mechanism to select relevant and important features and additionally refine them.
The closest methods to our work are [58,64,69]. There are several notable differences as compared to our method. First, these methods employ a sequence of channel-wise and spatial attention after every convolutional layer to refine the feature maps. In contrast, we specifically insert a spatial attention module after the conv3 block. By doing this, we are able to infuse the attention early into forward process and hence, such a module is rendered unnecessary after every block for an application like crowd counting. Second, the spatial attention modules in the existing works are selfsupervised. Different from them, we employ a mask guided spatial attention module that is explicitly supervised using foreground/background masks, resulting in faster learning of the spatial attention. Further, the global attention modules are inserted only after conv4 and conv5 blocks. These different modules are carefully added at specific blocks in the network, thereby avoiding unnecessary over-parameterization.
Weak Supervision. Weakly supervised learning has been extensively used for various problems in computer vision such as semantic segmentation [73,74,75,76,77,78], object localization [79,80,81,82], saliency detection [83,84], scene recognition [85,86] and many more. However, this form of learning has been relatively unexplored for crowd counting. Liu et al. [87] proposed a solution based on Bayesian model adaptation of Gaussian processes for transfer learning, which is limited only to the GP model of [88]. Recently, Borstel et al. [89] proposed a method to count objects within an image from only region-level count information. Though the problem is defined in a weakly supervised setting, they require local region level count as annotations which is a labour intensive process.
In this work, we introduce a novel weakly-supervised learning setup that employs image-level labels to generate pseudo ground-truth, which is further used to fine-tune the counting network. The use of pseudo ground-truth generation is inspired from semi-supervised learning approaches such as [90,91,92,93,94,95]. These approaches typically use labeled data in the dataset to train a predictor. The trained predictor is then used to generate predictions for the unlabeled data. The highly confident predictions are then used as ground-truth to further fine-tune the data. In contrast to this framework, we propose a weakly supervised approach where the pseudoground truth is generated from weak image-level labels.

III. HIERARCHICAL ATTENTION FOR CROWD COUNTING
As discussed earlier, a natural solution to address scale variation in crowd counting images is to leverage multiscale features from different layers in the backbone network. However, the layers in the backbone network are learned in a hierarchical manner with the earlier layers capturing primitive features and the subsequent layers capturing higher level concepts. Hence, direct fusion of these multi-scale feature maps might not be the most effective approach. In order to overcome this, we propose Hierarchical Attention-based Crowd Counting Network (HA-CCN) that leverages attention mechanisms to enrich features from different layers of the network for more effective multi-scale fusion. Fig. 3 provides an overview of the proposed method, which is based on the VGG-16 network. We include a spatial attention module (SAM) and a set of global attention modules (GAM) with the objective of enriching the feature maps at different levels. The base network consists of conv layers (conv1 ∼ conv5) from the VGG16 network. The conv3 features are enhanced by passing them through SAM. Similarly, features from conv4 and conv5 are passed through GAMs in order to perform channel-wise enhancement. The enhanced feature maps from conv3 are These processed features are concatenated together before being forwarded through the fusion module that consists of a set of conv layers to produce the final density map. These conv layers are defined by: {Conv2d(72,64,1) 3 -ReLU, Conv2d(64,64,3) 3 -ReLU, Conv2d(64,1,1) 3 -ReLU}. The network is trained by minimizing the Euclidean distance between the predicted density map and the ground truth density map as below: where, N is number of training samples, is the estimated density, D i is the i th ground-truth density and it is calculated by summing a 2D Gaussian kernel centered at every person's location x g as follows: , where σ is scale parameter of 2D Gaussian kernel and S is the set of all points where people are located. The density map generated by the network is 1/4 th of the input image resolution. Due to its construction, the sum of the density map provides an estimate of the number of people in the input image.
Details of the proposed method and its various components are described in the following sub-sections.

A. Spatial attention module
Inspired by the success of spatial attention mechanisms in image captioning, visual question answering and classification [58,62,68], we explore its utilization for crowd counting. The goal of spatial attention is to select attentive regions in the feature maps, which are then used to dynamically enhance the feature responses. In contrast to existing work that learn spatial attention in a self-supervised manner, we propose to learn this by explicitly using foreground background segmentation for supervision. Since the goal of the spatial attention module is to focus on relevant regions and foreground regions are necessarily a part of these relevant regions, it is beneficial to use these labels to supervise the module. By explicitly supervising the module, we are able to infuse foreground background information into the network, thereby forcing the network to focus on relevant regions among the foreground. Moreover, these labels are readily available and hence, it does not require additional annotation efforts.
The spatial attention module consists of 4 conv layers with 3×3 filters that takes feature maps from the conv3 layer of the VGG16 network as input (denoted by X ∈ R W ×H×C ), and produces a segmentation output S a ∈ [0, 1] W ×H . The segmentation map is then used to actuate the low level feature map X via element-wise multiplication:X = X S a , whereX is the actuated low level feature map from conv3. Through this attention mechanism, we are able to incorporate segmentation awareness into the low level feature maps. As illustrated in Fig. 4, the use of segmentation information into the network enriches feature maps by suppressing irrelevant regions and boosting the foreground regions. The actuated feature maps are then forwarded to the fusion block (FM), where they are fused with the features from other layers to generate the final density map. The weights of SAM are learned by minimizing the cross entropy error between the predicted segmentation map and the corresponding ground-truth. Normally, the segmentation task requires pixel-wise annotations. However, in this case existing ground truth density map annotations are thresholded to generate the ground truth segmentation maps, which are then used to train the spatial attention module. Basically, the pixels that contain head regions are labeled as 1 (foreground), and otherwise as 0 (background). Hence, the proposed method does not require any additional labeling. In spite of these annotations being noisy, the use of segmentation information results in considerable gains.

B. Global attention modules
In contrast to the spatial attention module that attends to relevant spatial locations in the feature maps of low-level layers, the global attention module (GAM) is designed to attend to feature maps in the channel-dimension. The global attention module is similar to the channel-wise attention used in earlier work like [58,64]. Specifically, this module consumes feature maps from the backbone network and learns to compute attention along the channel dimension. The computed attention captures the important channels in the feature maps and hence aids in suppressing information from unnecessary channels. Since this module operates at a global level in terms of spatial dimension, we refer to this attention module as global attention module. It has been demonstrated in [58,68,72], that channels capture the presence either different parts of an object or different classes of objects and channel-wise attention is an effective way to boost the correlation between object/object parts and image captions.
Based on these considerations, we employ a set of global attention modules, which take feature maps from the higher conv layers as input and produce a channel-wise attention map, which is then used to actuate the feature maps along the channel dimension. Mathematically, given a feature map input X ∈ R W ×H×C , GAM first performs a spatial pooling to produce pooled features Y ∈ R 1×1×C using where i is the channel index, and w, h are spatial indices. Y is passed through a set of fully-connected (FC) layers defined by F C(512, 64) − ReLU − F C(64, 64) − ReLU − F C(64, 512) 2 and a sigmoid layer to produce channel-wise attention vector S g ∈ R 1×1×C . Finally, S g is used to actuate the feature maps from the higher layer by performing a element-wise multiplication, i.e.,X = X S g .

IV. WEAK SUPERVISION VIA IMAGE-LEVEL LABELS
As discussed earlier, existing methods [1,2] recognize the inability of these networks to generalize well to different datasets. Their solutions to improve the cross-dataset performance is through fine-tuning in either a fully-supervised or semi-supervised fashion. In contrast to these approaches, we propose a weakly supervised setup to train the counting networks on the new datasets with just image-level labels. Such a setup will simplify the training process as it does not require point-wise annotations which are labour intensive and expensive to obtain. The idea of performing weakly supervised crowd counting is largely inspired by the success of recent CNN-based weakly supervised semantic segmentation methods [73,74,75] that typically fit the problem into Multiple-Instance Learning framework [96]. In their setup, every image is considered to have at least one pixel corresponding to image class label, and the segmentation task is formulated as inferring the pixels belonging to the object class. These methods usually employ class activation mappings to perform weak supervision. However, crowd counting is a regression problem and cannot be directly fit into such a framework. To overcome this issue, crowd counting is transformed into a crowd-density classification task, i.e., instead of counting the number of people in an image, this task is reformulated into categorizing the image into one of the six classes: C = {zero density, verylow density, low density, medium density, high density, veryhigh density}. This reformulation is based on the intuition that it is easier to label an image as containing large or few number of people as compared to the exact count. Sindagi et al. [4] used a similar concept for leveraging image context. In this work, the labels are used to reformulate the counting problem into a classification task for weakly supervised learning. Fig. 5 illustrates the proposed weakly supervised approach for adapting to new target scenes or datasets. Similar to semantic segmentation where a pre-trained CNN is used, we use counting network (HA-CCN) described in Section III that is pre-trained on the source dataset. A class activation map module (CAM), consisting of 4 conv layers is added before the fusion module in the counting network. This module is This sub-network takes in features from the counting network and processes them to produce output with |C| feature planes, one for each class. That is, the output of CAM is pixel-wise scores for each class and is denoted by S c i,j at pixel location (i, j) for each class c C. Since point-wise labels are not available for the target set, the pixel-wise scores for each class are mapped to a single image-level classification score using an aggregation function F agg such that s c = F agg (S c i,j ). This class-wise (s c ) score is then maximized for the right class label. Different aggregation functions such as Global Average Pooling (GAP) and Global Max Pooling (GMP) [97] have been used in the literature. In case of GAP, all pixels in the score map are assigned the same weights even if they do not belong to image's class label. GMP addresses this by assigning weight to the pixel that contributes most to the score, however the training is slow [75]. Hence, smooth version and convex approximation of the max function is chosen for F agg , called Log-Sum-Exp (LSE) which is defined as: where, S c denotes aggregated score for class c, S c i,j is pixellevel score at location (i, j) for class c, r is a hyper-parameter that controls the smoothness of approximation, w, h are width and height of the score map. A soft-max function is applied to the aggregated class scores. The CAM module is trained using the standard binary cross entropy loss function. Parameters of the counting network are kept fixed during this training. The class-wise score maps obtained from the above procedure indicate regions/pixels in the image that belong to a particular density level and hence can be viewed similar to class activation maps [98] (see Fig. 6). These class-wise maps are then used to approximate the pseudo ground-truth density maps for the target set using: where, S c i,j are obtained by normalizing S c i,j and n(c) is the average count for class c in the source dataset. The pseudo ground-truth maps (as seen in Fig. 6) are not as sharp as actual ground-truth maps, however, they provide coarse regional density that is better as compared to just image-level labels. Fig. 6. Example of class-wise score maps overlaid on input images. It can be observed that the CAM module is able to accurately identify regions corresponding to different density levels in an image. We also illustrate pseudo ground-truth estimated using image-level labels. Note that in the density maps, red color indicates high density and blue color indicates low density.
These pseudo ground-truth density maps are used to supervise the counting network on the target dataset. During finetuning, weights of the VGG-16 network are fixed and only the weights of the later conv layers are updated. This ensures that the resulting estimated density maps are sharper since the feature maps extracted from VGG-16 preserve details, while the later layers adapt to the newer dataset.
Although the network is trained using image-level labels, it learns to generate density maps for the target set as well. Hence, during inference, test image from the target set is forwarded through the network to estimate the density map. The performance of the proposed weakly supervised technique is measured using standard count error metrics (MAE/MSE).

A. Hierarchical attention-based counting
In this section, we first describe the training and implementation specifics followed by a detailed ablation study to understand the effects of different components in the proposed counting network. Finally, we compare results of the proposed method against several recent approaches on 3 publicly available datasets (ShanghaiTech [2], UCF-QNRF [42], UCF CROWD 50 [24]).

1) Training and implementation details:
The network is trained end-to-end using the Adam optimizer with a learning  rate of 0.00005 and a momentum of 0.9 on a single NVIDIA GPU Titan Xp. 10 % of the training set is set aside for validation purpose. The final training dataset is formed by cropping patches of size 224×224 from 9 random locations from each image. Further data augmentation is performed by randomly flipping the images (horizontally) and adding random noise. Similar to earlier work, the count performance is measured using mean absolute error (M AE) and mean squared error (M SE) given by: where N is the number of test samples, y i is the groundtruth count and y i is the estimated count corresponding to the i th sample.
Since the network is fully convolutional, entire test image is forwarded through the network during inference. This results in faster inference as compared to the existing methods (such as Switching-CNN [3], IG-CNN [8], CP-CNN [4], SA-Net [9]) which involve patch-based testing.
2) Architecture ablation: To understand the effectiveness of various modules present in the network, we perform experiments with the different settings using ShanghaiTech dataset (Part A and Part B). This dataset consists of 2 parts with Part A containing 482 images and Part B containing 716 images and a total of 330,165 head annotations. Both parts have training and test subsets.
The ablation study consisted of evaluating 3 baselines in addition to the proposed method: (i)VGG16: VGG16 network with an additional conv block at the end. (ii)VGG16+MS: VGG16 with multi-scale feature map  [42]. The results of these experiments are tabulated in Table I. It can be observed that the naive approach of performing multiscale feature concatenation does not necessarily yield the most optimal performance. The use of SAM infuses segmentation information in to the feature maps of conv3 layer in the base network, resulting in considerable reduction of the count error as compared to the naive approach. The use of global attention results in further improvement, thus showing significance of incorporating channel-wise importance in the network.
Additionally, it can also be noted that the explicitly supervised SAM results in better performance as compared to the self-supervised spatial attention. Fig. 7 shows a plot of the mean absolute error for different configurations in the ablation study at different density levels. It can be observed that the proposed HA-CCN network achieves best error among all the density levels.
3) Comparison with recent methods: In this section, we discuss the results of the proposed method as compared with recent approaches on 3 different datasets: ShanghaiTech [2], UCF CROWD 50 [24] and UCF-QNRF [42]. As discussed earlier, ShanghaiTech has 2 parts with a total of 1198 images. The UCF CC 50 dataset [24] contains 50 annotated images of different resolutions and aspect ratios. Following the standard protocol discussed in [24], a 5-fold cross-validation is performed for evaluating the proposed method. UCF-QNRF [42] is a more recent dataset that contains 1,535 high quality images with a total of 1.25 million annotations. The training and test sets consists of 1201 and 334 images respectively. Table II shows the results of the proposed method on the ShanghaiTech and UCF CROWD 50 datasets as compared with several recent approaches: Switching-CNN [3], CP-CNN [4], IG-CNN [8], D-ConvNet [6], Liu et al. [5], CSRNet [41], ic-CNN [10], SA-Net [9], ADCrowdNet [44] and Residual Regression [45]. It can be observed that the proposed method outperforms all existing methods. Table III shows the comparison of results on the recently released large-scale UCF-QNRF [42] dataset. The proposed method is compared against five different approaches: Idrees   [42]. It can be observed that the proposed method is able to achieve state-of-the-art results on this complex dataset. Fig. 8, 9 and 10 illustrate the qualitative results for sample images from the ShanghaiTech, UCF CROWD 50 and UCF-QNFRF datasets respectively.

B. Cross dataset performance
We compare the generalization abilities of the proposed method with that of recent methods (MCNN [2], Switching-CNN [3], D-ConvNet [6]) by testing the network (trained on ShanghaiTech A dataset) on target datasets such as Shang-haiTech B, UCF CROWD 50 and WorldExpo '10 [1]. The results are presented in Table IV. Note that the other networks are also trained on ShanghaiTech A dataset. The crossdataset performance is measured using the overall count error (MAE/MSE) and the drop in performance. The drop in performance is the difference between the error of the model trained on the target set and that of the model trained on source set, when tested on target set. It can be observed that the proposed method is relatively more robust to change in dataset distribution as compared to the other methods.
Although the proposed method demonstrates better crossdataset performance as compared to existing methods, there is considerable gap in the performance as compared to when the network is fully supervised on the target set. We address this issue via the weakly supervised technique described in Section IV.

C. Weakly supervised counting
In this section, we present the experiment details and results of weak supervision setup. Training. First, a source training set is created that is based on the ShanghaiTech A dataset. The other datasets (ShanghaiTech B, UCF CROWD 50 and WorldExpo) are used as the target sets. ShanghaiTech A is chosen for creating the source training set since it contains large variations in density, scale and appearance of people across images. The training set is created by cropping multi-scale patches of size 224×224 from 9 random locations. The multi-scale patch extractions increases diversity of the source dataset in terms of count and field of view. Image-level labels for the source dataset are assigned based on the count in each image in the source set.
The target training set is created by cropping multi-scale patches from 9 random locations from each image. The imagelevel labels for the target set are obtained based on the count in each image. To compensate for the fact that count values from the target set are used to obtain the image-level labels (which is not practically feasible since the target set is not supposed to have point-wise or count annotations), label noise is added for 15% of the training samples. That is, we randomly changed the labels of 15% of the samples with the neighboring classes. This process of adding label noise simulates human labeling error.
The crowd counting network is first trained on the diverse source dataset using full-supervision by minimizing the loss function described in (1), followed by addition of the CAM module. Weights of the counting network are fixed and the CAM module is trained on the diverse source dataset by minimizing the binary cross entropy between image-level labels and aggregated class scores. This is followed by finetuning of the CAM module on the target samples using imagelevel labels. The class-wise maps from the CAM module are used to generate the pseudo ground-truth density maps for the target samples which are then used to fine-tune the counting network.
Discussion. The results of adapting pre-trained counting model using weak supervision and selective fine-tuning for three target datasets (ShanghaiTech B, UCF CROWD 50 and WorldExpo) are shown in Table V It can be observed that the proposed WSL setup results in significant improvements in the generalization performance of the network. Among the three aggregation functions for weakly supervised learning, LSE outperforms the other two functions. The results obtained using WSL are comparable to many recent fully supervised techniques such as Hydra-CNN [33], MCNN [2], Walach et al. [32], Switching-CNN [3], thus demonstrating the significance of the proposed weak supervision technique.

VI. CONCLUSIONS
In this work, we presented a crowd counting network that consists of different attention mechanisms at various levels in the network. Specifically, the proposed network involves two sets of attention modules: spatial attention and global attention module. The spatial attention module incorporates pixel level attention through a way of foreground background segmentation into the features of the earlier layers of the network. The global attention module incorporates channel-wise importance into the network. Furthermore, we also presented a novel weakly supervised setup to adapt counting models to different datasets using image-level labels. Extensive experiments performed on challenging datasets and comparison with recent state-of-the-art approaches demonstrated the significant improvements achieved by the proposed method.
In the future, we will explore better ways of incorporating features from different layers and extend the current framework to other backbone networks. Additionally, we will explore other forms of weakly supervised and semi-supervised learning approaches to further improve the cross-dataset performance.
ACKNOWLEDGMENT This work was supported by US Office of Naval Research (ONR) Grant YIP N00014-16-1-3134.