extfieldsofstudy:
Computer Science
Physics

s2fieldsofstudy:
Computer Science
Physics

text:
Surveying the reach and maturity of machine learning and artificial intelligence in astronomy

Machine learning (automated processes that learn by example in order to classify, predict, discover or generate new data) and artificial intelligence (methods by which a computer makes decisions or discoveries that would usually require human intelligence) are now firmly established in astronomy. Every week, new applications of machine learning and artificial intelligence are added to a growing corpus of work. Random forests, support vector machines, and neural networks (artificial, deep, and convolutional) are now having a genuine impact for applications as diverse as discovering extrasolar planets, transient objects, quasars, and gravitationally-lensed systems, forecasting solar activity, and distinguishing between signals and instrumental effects in gravitational wave astronomy. This review surveys contemporary, published literature on machine learning and artificial intelligence in astronomy and astrophysics. Applications span seven main categories of activity: classification, regression, clustering, forecasting, generation, discovery, and the development of new scientific insight. These categories form the basis of a hierarchy of maturity, as the use of machine learning and artificial intelligence emerges, progresses or becomes established.

Introduction
Astronomy has a rich history of data gathering and record keeping [Brunner et al., 2002, Feigelson and Babu, 2012, Jaschek, 1968, 1978, Zhang and Zhao, 2015. Data about and from celestial objects is collected using an assortment of telescopes, photon detectors and particle detectors. While all of the electromagnetic spectrum is of interest, the bulk of observational data comes from the visible/infrared (wavelengths from 400 nm to 1 mm) and radio (wavelengths from 1 cm to 1 km) portions of the spectrum. Much of this data is recorded in the form of two-dimensional pixel-based images and one-dimensional spectra. Secondary data products are derived from observational data, often as catalogues of individual source properties: position, size, mass, chemical composition, and so forth. Observational data can be recorded and analyzed at a single epoch, or the properties of astronomical sources -especially brightness and position -can be monitored over time.
Complementing observational data gathering are the dual fields of numerical simulation and astrophysical theory, although there is a great deal of overlap between the two. While some branches of theory do not in themselves produce large quantities of data, focusing instead on mathematical descriptions of cosmic phenomena, computer simulations generate data that can be used to model, predict, and support analysis of the observational data.
Modern astronomical data is measured in Terabytes, Petabytes and, soon, Exabytes. When astronomy crossed the 100 Terabyte scale near the end of the 20th century [e.g., Brunner et al., 2002, Szalay andGray, 2001] a new data-driven astronomy emerged: where data mining was mooted as the likely future [Bell et al., 2009, Ivezić et al., 2014, Szalay and Gray, 2006. The expectation was an increased reliance on automated systems to locate, classify, and characterise objects. At the same time, new fundamental relationships between derived properties might be found, by allowing clever algorithms to search through complex, multi-dimensional data catalogues [e.g. Graham et al., 2013].

Data mining
The background and early history of data mining in astronomy is covered in some detail by Borne [2009], Ball and Brunner [2010], and the collection of articles in Way et al. [2012]. An early emphasis of data mining was to find new samples of rare sources, by applying workflows that gathered data from large, often online, repositories. Lépine and Shara [2005] applied a software blink comparator to 615,800 sub-fields downloaded from the Digitized Sky Survey, identifying and cataloging 61,977 stars with high proper motions (Dec ¿ 0 • ). Targeting the planned SkyMapper [Keller et al., 2007] Southern Sky Survey, Walsh et al. [2007] mined the Sloan Digitzed Sky Survey (SDSS) Data Release 5 [DR5; Adelman-McCarthy et al., 2006] for stellar over-densities, uncovering a new Milky Way dwarf galaxy satellite (Boötes II). González-Solares et al. [2008] opened up their INT/WFC Photometric Hα Survey of the Northern Galactic Plane (IPHAS) dataset through the AstroGrid Virtual Observatory Desktop, 1 with the goal of making their 200 million-object photometric catalog available for data mining. Discoveries resulting from exploration of the IPHAS initiative included new samples of young stars , planetary nebulae [Viironen et al., 2009], and galactic supernova remnants [Sabin et al., 2013]. Virtual Observatory infrastructure was also utilized by Chilingarian et al. [2009] in a workflow to identify a sample of compact elliptical galaxies. Candidates were selected by leveraging a combination of resources including imaging data from the VizieR Catalogue Service 2 at the Centre de Donnes Astronomiques de Strasbourg, the NASA/IPAC (National Aeronautics and Space Administration/Infrared Processing and Analysis Center) Extragalactic Database (NED) 3 , the Hubble Legacy Archive 4 , and photometric and spectroscopic results from SDSS Data Release 7 [DR7; Abazajian and et al., 2009]. SDSS catalogues also played a role in projects such as the identification of dwarf novae candidates [Wils et al., 2010], found by cross-matching DR7 with an astrometric catalogue from the Galaxy Evolution Explorer (GALEX) space mission [Martin et al., 2005]. The need for practical data mining infrastructure led to the development of tools such as DAMEWARE [DAta Mining & Exploration Web Application and REsource; Brescia et al., 2014Brescia et al., , 2016. See also Ivezić et al. [2014] and the AstroML Python module 5 for a selection of data mining implementations, with an emphasise on large-scale observational surveys.

The emergence of machine learning and artificial intelligence in astronomy
The value of automated data mining as an approach to knowledge discovery in astronomy has been firmly established across a broad range of sub-disciplines of astronomical interest. Within many fields of astronomy, though, the discussion of data mining is evolving rapidly to focus almost exclusively on machine learning (ML; automated processes that learn by example in order to classify, predict, discover or generate new data) and, to a lesser extent, artificial intelligence (AI; methods by which a computer makes decisions or discoveries that would usually require human intelligence). The developing use of ML and AI in astronomy has mirrored the broader use in computer science and the scientific community. Traditional statistical techniques found application first. Principal component analysis (PCA) was used, for instance: in the 1980s for morphological classification of spiral galaxies [Whitmore, 1984]; in the 1990s for quasar detection [Francis et al., 1992] and stellar spectral classification [Singh et al., 1998]; and in the 2000s for galaxy classification [Conselice, 2006] and quasar detection in the Sloan Digital Sky Survey [Yip et al., 2004]. PCA is now a standard technique, which continues to be used in hundreds of astronomy projects and papers per year.
By the early 1990s, astronomers begin to take advantage of more complex methods requiring labelled training sets. In the 1990s, decision Trees (DTs) began to be employed for tasks such as star-galaxy separation [Weir et al., 1995] and galaxy morphology classification [Kriessler et al., 1998, Owens et al., 1996. By the 2000s, use of the technique proliferated and random forests (RFs) begin to dominate, with a key application being photometric redshift estimation [Carrasco Kind and Brunner, 2013]. Boosted decision tree techniques, such as AdaBoost, appeared in more recent years and continue to be used, including for assignment of photometric redshifts [Hoyle et al., 2015a] and for star-galaxy separation [Sevilla-Noarbe and Etayo-Sotos, 2015]. Support Vector Machines (SVMs) also found application in the 2000s and beyond, for instance in the detection of red variable stars [Woźniak et al., 2004], determination of photometric redshifts [Wadadekar, 2004], prediction of solar flares [Qahwaji and Colak, 2007], star-galaxy separation [Fadely et al., 2012], and noise analysis in gravitational wave detection [Biswas et al., 2013].
One of the dominant machine learning techniques, the artificial neural networks (ANN), appeared in the field at the end of the 1980s [Angel et al., 1990, Rosenthal, 1988 and by the 1990s was applied across a broad range of problems in astronomy. Early applications included star-galaxy separation [Odewahn et al., 1992], galaxy morphology classification [Lahav et al., 1996, Storrie-Lombardi et al., 1992, and object detection in the staple astronomical software Source Extractor [SExtractor; Bertin and Arnouts, 1996]. By the 2000s, ANNs were playing a key role in photometric redshift estimation [Collister and Lahav, 2004, Firth et al., 2003, Vanzella et al., 2004, galaxy classification [Ball et al., 2004] and the detection of gamma ray bursts [GRBs; Ball et al., 2004]. Paving the way for the "Deep Learning" era, the use of ANNs in astronomy has accelerated over the last decade, for instance in the analysis of asteroid composition [de León et al., 2010], pulsar detection [Eatough et al., 2010], and finding gravitationally lensed quasars [Agnello et al., 2015].
Two strongly linked occurrences have had a significant impact on the growth of adoption of ML and AI in astronomy. First was the appearance of graphics processing units (GPUs) as affordable, massively parallel computational accelerators, with applicability to a wide range of computationally-demanding problems [see, for example, Barsdell et al. [2010], Fluke et al. [2011] for adoption strategies in astronomy]. Secondly was the emergence of deep neural networks and convolutional neural networks. These approaches -extensions to the "vanilla" ANN -benefit from GPU acceleration to perform computationally arduous calculations in parallel in a reasonable time and at relatively low cost. For data-rich fields, such as astronomy, the predictive performance of these deep learning networks improves as more data is provided for training and tuning.
In the field of computer vision -the computational analysis of image data -the use of deep neural networks also accelerated after the spectacular demonstration by Krizhevsky et al. [2012] of the power of convolution neural networks applied to classifying images of millions of everyday objects. Astronomers were quick to take advantage of this revolution, with Dieleman et al. [2015] and Huertas-Company et al. [2015] achieving human-level performance on galaxy morphology classification and  demonstrating the possibility of estimating photometric redshifts directly from images using convolutional neural networks.

Scope and structure
This advanced review surveys progress in the wide-scale adoption of machine learning and artificial intelligence within astronomy, as evidenced by a collection of recently published works. Techniques are not explained and referenced in detail, expect with respect to their particular adoption in astronomy. Most advances in astronomical and astrophysical knowledge have relied on a relatively small number of general methods (see Section 2).
The primary avenue for identifying relevant literature was NASA's Astrophysics Data System 6 [Chyla et al., 2015, Kurtz et al., 2000] -a knowledge discovery tool without equal. To gather an extensive and representative current sample of publications, abstracts of published, peer reviewed journal articles were probed for key words such as "machine learning", "artificial intelligence", "neural networks", and "data mining". Bayesian methods are intentionally omitted, as these align more naturally with traditional statistical methods [Heck et al., 1985]. Such an approach does miss important research results, so no claim is made as to the exhaustiveness or completeness of the review. Progress in the adoption of ML and AI in astronomy is occurring rapidly. However, through the qualitative examination of ∼ 200 refereed publications from 2017 to February 2019, using an approach sharing elements with Grounded Theory, 7 a broad collection of astronomy applications has been assessed such that common themes have emerged regarding the reach and maturity of ML and AI in astronomy.
Indicative examples are drawn from the recent published literature to highlight how ML and AI techniques are used across seven categories of activity (Section 2) and three phases of maturity (Section 3). It is important to remember that a successful use of a machine learning algorithm is more likely to be reported than one where a method failed to work. Counter examples or cautionary tales [e.g. Connor and van Leeuwen, 2018], where an algorithm may not have performed as well as hoped, are rare.

Machine learning and artificial intelligence in astronomy
Data-driven scientific discovery occurs through a combination of statistical methods, machine learning and artificial intelligence techniques, and the use of database systems. 8 Scientific discovery requires techniques for identifying patterns within datasets (the original scope of data mining) as part of a multi-stage process for selecting, cleaning, processing, and transforming raw data into useful knowledge (i.e., knowledge discovery in databases, as described in Fayyad et al. [1996]).
As highlighted in Section 1, global astronomy data collections are approaching the Exabyte scale. A key motivation for many applications of ML and AI to astronomical data is the need to prepare for the data streams expected from near-term observatories and space missions. The Large Synoptic Survey Telescope [Ivezić andet al., 2019, LSST Science Collaboration and; the Euclid satellite [Laureijs andet al., 2011, Amendola et al., 2013]; MeerKAT [Booth et al., 2009]; the Australian Square Kilometre Array Pathfinder [Johnston et al., 2007[Johnston et al., , 2008; and the Square Kilometre Array [Dewdney et al., 2009], among others, will all generate datasets on scales (volumes and velocities) that vastly exceed the 6 http://ui.adsabs.harvard.edu 7 Grounded Theory is a qualitative analysis strategy, used, for example, in education research and social sciences, where a sequence of reader-assigned codes is applied to allow identification and tracking of themes within a sample of relevant literature. Codes do not have to be selected in advance, but are defined dynamically in multiple iterations through the literature.
8 As proposed by Ball and Brunner [2010], a sufficiently flexible definition for a database in astronomy is "any machinereadable astronomical data". discovery capabilities of humans. In the interim, the Sloan Digital Sky Survey [SDDS; York and et al., 2000, Stoughton and et al., 2002, Abazajian and et al., 2009, the Panoramaic Survey Telescope and Rapid Response System [Pan-STARRS; Kaiser, 2004], the Catalina Real-Time Transient Survey [CRTS; , Mahabal et al., 2011 [Iodice et al., 2016], both using the the VLT Survey Telescope 9 , LOFAR [van Haarlem and et 2013], the Solar Dynamic Observatory [SDO; Lemen et al., 2012, Pesnell et al., 2012, the Kepler Planet-Detection Mission [Borucki and et al., 2010], and the GAIA space mission et al., 2016b,a, 2018], are generating data with which ML and AI has enabled classification, regression, forecasting, and discovery, leading to new knowledge and new insights.

The nature of the data
The applicability and efficacy of any ML or AI technique depends on the nature of the data. Brunner et al. [2002] classified astronomical data into five domains, extended slightly here to allow a clearer connection to the specific applications of ML and AI within astronomy. In this section, references are given to recent works that apply ML and AI to each of the data types, rather than to the originator(s) of the data type.
Images are pictures of astronomical objects (usually as a pixel grid of numerical intensity values), such that the appearance informs a classification [e.g., Aniyan and Thorat, 2017, Xin et al., 2017, Kuminski and Shamir, 2018 or provides insight about physical processes that are occurring . For visible/infrared (IR) observations (i.e. optical astronomy), light passes through, and is focused by, a telescope's optical system to be captured on a charge-coupled device (CCD). Various filters are used to select only specific regions of the visible/IR spectrum. For radio observations, it is common to refer to the frequency bandwidth over which flux is recorded from a particular location in the sky, with most radio images created from interferometers using the technique of aperture synthesis.
Spectroscopy refers to wavelength-dependent numerical intensity measurements over a finite range of wavelengths (or frequencies), but often with very high resolution. Spectroscopy provides information on the atomic and molecular composition, from which other physical properties (temperature, density, metallicity, etc.) can be inferred [e.g., , Márquez-Neila et al., 2018, Miettinen, 2018, Ucci et al., 2018.
A special case at the intersection of imaging and spectroscopy is the spectral cube [e.g., Araya et al., 2018, Bron et al., 2018. This is a volumetric dataset comprising a sequence of images, each captured over a very narrow wavelength or frequency range. When looking for structures within a spectral cube, it is treated as an image. When extracting a spectrum at a fixed spatial location, it is treated as a spectroscopic data product. Spectroscopic data cubes, with their high dimensionality, may prove a challenge for established machine learning methods to handle. Convolutional neural networks have proved to work robustly on astronomical image data in several photometric bands, so there is no theoretical obstacle to extending this to thousands of spectroscopic frequencies. However, the practical challenges are yet to be fully explored.
Photometry is concerned with accurate measurements of the brightness (i.e. intensity, luminosity, flux) of an object recorded through a filter. It is a secondary, numerical data product derived from a calibrated image. Comparisons between photometric measurements through different filters are often used as an alternative to detailed spectroscopic observations, with specific application to determining the distance to a celestial source [e.g. Cavuoti et al. [2017a], Morrison et al. [2017], Beck et al. [2018], Bilicki et al. [2018]].
Images, spectroscopic, and photometric measurements can be made as a function of time. For optical astronomers, a light curve is time-based photometry, where the variation in intensity of a source over time helps with the identification and classification of a variety of variable star types [e.g. Cohen et al., 2017, Naul et al., 2018, Papageorgiou et al., 2018 or indicates the presence of otherwise unseen objects, such as an extrasolar planet [e.g. Mislis et al., 2018, Pearson et al., 2018, Shallue and Vanderburg, 2018. In this review, other time-based measurements in radio astronomy [e.g. pulsar and transient object searches - Connor and van Leeuwen [2018], , Pang et al. [2018], , Farah et al. [2018]], and the emerging field of observational gravitational wave astronomy [Powell et al., 2017, Zevin et al., 2017, George and Huerta, 2018b, are categorized as time series.
The end product of many data gathering programs is a catalogue, which comprises one or more numerical or categorical data types. Some may be derived from the standard data gathering approaches introduced above, while others are calculated or otherwise derived -including through ML [e.g. Marchetti et al., 2017, Tachibana and. Within a catalogue, astrometry refers to the accurate measurement of the spatial locations of objects,on the celestial sphere or with respect to an alternative coordinate system [Castro-Ginard et al., 2018, Gao, 2018a. Most objects are reported with at least one measurement of position. Some local objects, such as the Solar System's planets and minor planets, and the growing number of stars within the reach of the GAIA space mission, move with respect to the coordinate system over time , ]. A morphological classification places a particular type of object into an object-based category where a common physical process (or set of processes) is thought to drive the appearance of an object [e.g. Aniyan and Thorat, 2017, Kuminski and Shamir, 2018.
Finally, although a single catch-all name does not do such a diverse field justice, simulation will be used to describe the data products from any numerical or computational method. For example, cosmological simulations [e.g. Agarwal et al., 2018, Hui et al., 2018, Lucie-Smith et al., 2018, Nadler et al., 2018, Rodríguez et al., 2018 follow the gravity-induced formation and growth of structures, requiring approximations to various physical mechanisms, a suitable choice of initial conditions, and a strategy for time-based evolution (down to some minimum level of accuracy).

From classification to insight
For the science of astronomy to progress through the use of ML and AI, it must be possible to demonstrate that the outcome is not merely an automated classification or a numerical prediction, but that astronomers are using the data mining phase to discover new objects or generate new insights into the underlying physical processes and relationships. In assessing the recently published literature, seven categories emerge pertaining to how ML and AI are used in astronomy (cf. Fayyad et al. [1996] and Zhang and Zhao [2015], who identify similar categories). The last two categories (discovery and insight) are those where a higher order scientific outcome arises.
1. Classification: Categories or labels are applied to objects or features. Based on a training set (labelled or unlabelled), the machine learning algorithm learns the characteristics that relate an instance to a category. When applied to a new instance, the algorithm assigns the most likely category label.
2. Regression: Assignment of a numerical value (or values) based on the characteristics that are learnt or otherwise predicted by the machine learning algorithm. As with classification, a training set may be used or the characteristics may be inferred from the dataset.
3. Clustering: These algorithms determine whether an object or a feature is part of (i.e. a member of) something. This might be a physical structure or association -as in the more familiar usage of the term in astronomy as applied to open, globular, or galactic clusters -or a region within an N-dimensional parameter space.

Forecasting:
The purpose of the machine learning algorithm is to learn from previous events, and predict or forecast that a similar event is going to occur. There is an implicit time-dependence to the prediction.

Generation and Reconstruction:
Missing information is created, expected to be consistent with the underlying truth. The cause of the missing information might be due to the presence of noise, processing artefacts, or additional astronomical phenomena, all of which conspire to obscure the required signal.
6. Discovery: New celestial objects, features or relationships are identified as a consequence of the application of a ML or AI method.
7. Insight: Moving beyond the discovery of celestial objects, new scientific knowledge is demonstrated as a consequence of applying machine learning or AI. This includes cases where insight is gained into the suitability of applying machine learning, choice of data set, hyperparameters, and comparisons with human-based classification.
Classification, regression and clustering processes are often presented as a comparison with a similar human-centred approach -but with a need to "scale-up" in terms of either the size of the dataset to be explored or "speed-up" the time taken to achieve the task. Classification and regression outcomes can either be the end-point of an investigation, or the input to a forecasting, generation, discovery or insight process. In Section 3, these categories will be used to make an assessment of the maturity of adoption of ML and AI in various sub-fields of astronomy.
A subset of discovery is the field of anomaly or outlier detection. Many of the most exciting discoveries to come are likely to lie among the "unknown unknowns" in new areas of parameter space captured within the Petascale and Exascale datasets of the future. New methods are being developed to find anomalous objects in astronomical datasets, such as the work by Baron and Poznanski [2016] using an unsupervised Random Forest to find outliers amongst SDSS galaxies. Promising avenues involve a combination of unsupervised learning methods, such as isolation forests [Liu et al., 2008]; dimensionality reduction, such as PCA, t-Distributed Stochastic Neighbor Embedding [t-SNE; van der Maaten and Hinton, 2008], e.g.  and Nakoneczny et al. [2019], self-organizing maps [SOMs; Kohonen, 1990], e.g. Carrasco Kind and Brunner [2014] and Armstrong et al. [2016], or the latent space of a variational encoder [e.g. Li, 2015, Ma et al., 2019]. Novel visualization techniques are also contributing. For instance, Masters et al. [2015], using self-organizing maps to vizualize the distribution of galaxies in photometric color space, were able to identify regions that were under-sampled spectroscopically and develop an optimal strategy for the Euclid mission's photometric redshift calibration efforts.
Based on a qualitative examination of a sample of ∼ 200 refereed publications from 2017 to February 2019, Table 1 summarises the mapping between the most common astronomical data types with the seven categories of ML/AI methods. Classification and regression algorithms are being applied to all of the data types. Although clustering activities span a number of data types, overall they were not common outside of studies of stellar clusters [Castro-Ginard et al., 2018, Gao, 2018a or segmentation processes [Bron et al., 2018.
Forecasting outcomes were mostly confined to images [e.g. Mukkavilli et al. [2018] with Mars images; , Nishizuka et al. [2017], Inceoglu et al. [2018], and Florios et al. [2018] using Solar magnetograms], photometric measurements [French and Zabludoff [2018] predicted likely tidal disruption events in post-starburst galaxies using a random forest algorithm] and catalogue data [forecasts of coronal mass ejections from the Sun based on ∼ 180 similar events using a support vector machine (SVM) ]. Generation methods, in particular generative adversarial networks, have been used with images from observations [Vavilova et al., 2018], and to simplify or remove the need for expensive numerical simulation [e.g. Diakogiannis et al., 2019, Rodríguez et al., 2018, Fussell and Moews, 2019.
With regards to the role of ML and AI in advancing knowledge in astronomy, there was clear evidence from the sample of recent publications that discovery tasks are being performed with all of the data types: images [Hartley et al., 2017, Ciuca and Hernández, 2017, Gomez Gonzalez et al., 2018 Table 1: From a qualitative examination of a sample of ∼ 200 refereed publications from 2017 to February 2019, a mapping emerges between the nature of astronomical data and the way that machine learning and artificial intelligence is actively been pursued. The table presents a qualitative summary of the categories of ML/AI algorithms and the most common types of astronomical data in the sample of publications.

Nature/Type
Classification Regression Clustering Forecasting Generation Discovery Insight ; light-curves [Armstrong et al., 2018, Peña et al., 2018, Cohen et al., 2017, Hedges et al., 2018, Heinze et al., 2018, van Roestel et al., 2018, Giles and Walkowicz, 2019; time-series , Pang et al., 2018, Connor and van Leeuwen, 2018, Farah et al., 2018, Morello et al., 2019; catalogues [Nguyen et al., 2018, Marchetti et al., 2017; and simulation: [Xu and Offner, 2017, Nadler et al., 2018, Kuntzer and Courbin, 2017. For most data types, outcomes have progressed to the insight phase, for example: enhanced understanding of human biases in classification of Galaxy Zoo project images ; determination of the evolution of the effective radius and stellar mass of Kilo Degree Survey [KiDS; de Jong et al., 2013] galaxies based on photometric redshifts derived from ML [Roy et al., 2018]; and new relationships between physical and envrionmental properties of galaxies by applying an SVM to the results of a cosmological simulation [Hui et al., 2018].

Techniques
Machine learning algorithms are usually classified as being either supervised or unsupervised. Supervised methods rely on a pre-labeled dataset, which is used to help train and tune the algorithm. This learning allows for new instances to be assigned a label (classification) or numerical value (regression). Unsupervised methods allow the data to speak for itself, but do not necessarily make use of any existing knowledge. Although there is no shortage of data in astronomy, there is often a paucity of relevant pre-labeled data. For example, when the discovery of rare events is the target of an observational program, it is very difficult to train a network on sufficient examples and counter-examples. Moreover, astronomical discovery does rely on serendipity -anomalous cases that are potentially unlike anything that has previously been examined, and hence no exemplars exist [Norris, 2017].
The lion's share of machine learning in astronomy is performed with five classes of algorithms: artificial neural networks; convolutional neural networks; decision trees; random forests; and support vector machines. These are primarily used as supervised learning algorithms. Since the Ball and Brunner [2010] review, convolutional neural networks are the only new method amongst these five to emerge and reach wide-spread usage in astronomy.
Artificial neural networks [ANNs; Rosenblatt, 1957, Fukushima, 1980 are the key technique behind the recent AI boom, but date back to the 1950s. They are designed by analogy to a biological neuron, with signals from multiple inputs weighted and added together; a biological neuron sends an electrical signal if these weighted input signals cross a certain activation threshold. In the case of an artificial neuron ("perceptron"), the inputs and trainable weights are vectors of real valued numbers, and the output is a scalar value. A single artificial neuron can be employed as a classifier, however they are typically combined in ANNs where the outputs of an array (layer) of neurons form the inputs to a subsequent layer. At the output layer, the values of one or more neurons are interpreted according to the problem domain. Optimization of the weights using a labelled training set follows the gradient descent paradigm with the backpropagation algorithm [LeCun et al., 1989].
Support vector machines [SVMs; Cortes and Vapnik, 1995], similarly to ANNs, learn non-linear decision boundaries in spaces of arbitrary dimension, finding a hyperplane in a space of arbitrary dimensionality that distinctly separates the supplied data. The algorithm works by finding the hyperplane with the maximum separation between extreme examples (support vectors). A linear SVM (LSVM) finds the optimal hyperplane for the supplied input features, but using the so-called 'kernel trick' a SVM projects the data into higher dimensions where the data is linearly separable.
Decision trees [DTs; Quinlan, 1986] perform classification by recursive binary splitting of the data, learning through the training process a series of decisions based on features of the input data. The root node (the entire data space) is repeatedly split into two child nodes based on the most discriminative feature of the data, until at the leaf node a category is determined. Random forests [RFs; Ho, T.K., 1995, Breiman, 2001 are an extension of decision trees, improving accuracy by constructing an ensemble of decision trees, trained on subsets of the training data [bagging; Breiman, 1996] and/or feature set (feature randomness), and using the median or mode of the ensemble as the final output value. AdaBoost [Freund and Schapire, 1995] is another ensemble method that weights the contribution of each decision tree based on misclassifications; similarly, gradient boosting [Friedman, 2001] uses decision trees trained in sequence on the residual errors of other DTs.
A convolutional neural network (CNN) is an extension of the simple ANN but with many hidden layers (i.e. a deep neural network). CNNs are characterized by their use of convolutional layers, which are sensitive to specific features -usually within images -that may have undergone transformations through translation, rotation, or scaling. Working in conjunction with pooling layers, which reduce the spatial size of image features within the network, the final stage of a CNN is often a fully-connected ANN to generate a classification or numerical prediction.
CNNs have now been used in astronomy for a variety of image-based classification, regression and discovery activities. They appear in the literature as: binary classifiers , Jacobs et al. , where there are multiple categories that have been determined previously, usually by human inspection, but also using other machine learning approaches [Kim and Brunner, 2017]; and for detection, with applications to discovery of exoplanet candidates in light-curves [Pearson et al., 2018], or real-time discovery of transient objects [Connor and van Leeuwen, 2018] and gravitational wave events [George and Huerta, 2018b,a].
Outside of this core group, are several unsupervised methods: k-nearest neighbours (k-NN), k-means clustering, and the DBSCAN method. The latter has been used as a discovery tool to improve efficiency at detecting exoplanet transits from light curves, based on the recovery of artificial transit signatures [Mislis et al., 2018], and in partnership with an ANN to identify open clusters [Castro-Ginard et al., 2018] in the GAIA DR2 [Gaia Collaboration and et al., 2018] -an example of a clustering process.
Other techniques that have been investigated, often in conjunction with one or more the above methods, include: AdaBoost [Xin et al., 2017, Bethapudi and; genetic algorithms [Sarro et al., 2018]; self-organizing maps [Armstrong et al., 2017, Süveges et al., 2017, Armstrong et al., 2018; recurrent neural networks [Naul et al., 2018]; auto-encoders [Vincent et al., 2008, Sedaghat andMahabal, 2018]; and transfer learning [Benavente et al., 2017]. Falling within the generation and reconstruction category (Section 2.2), generative adversarial networks (GANs) are likely to be the next most significant machine Table 2: From a qualitative examination of a sample of ∼ 200 refereed publications from 2017 to February 2019, a mapping emerges between the nature of astronomical data and the types of machine learning and artificial intelligence algorithms that are being applied. The table presents a summary of the types of astronomical data and the algorithms that appeared most regularly. The purpose of the table is to provide a convenient starting point for selecting an algorithm that has been used successfully for each data type.  Table 2 summarises the relationships found between the main types of astronomical data and specific techniques in the sample of recently-published papers. It is expected that specific techniques have been applied to other data types, and it is important to remember that some fields will have trialled particular methods and moved on as new alternatives appear. The Table's purpose is to emphasise areas of current activity and interest only, and thus provides a starting point for astronomers wishing to adopt a ML/AI approach by matching the data types to the methods.

Data/Method ANN CNN GAN SVM DT RF DBSCAN k-NN k-M
ANNs [Ciuca and Hernández, 2017, Marchetti et al., 2017, Bethapudi and Desai, 2018, Bilicki et al., 2018, Fujimoto et al., 2018, Ho, 2019, random forest methods [Schindler et al., 2017, Goulding et al., 2018, Hedges et al., 2018, Pang et al., 2018, Tachibana and Miller, 2018, Nadler et al., 2018, and SVM algorithms [Hartley et al., 2017, Hui et al., 2018, Kong et al., 2018 have been used extensively across most data types. CNNs are more suitable for image-style data (see above), although they have been used successfully with one-dimensional light curves [Shallue and Vanderburg, 2018, identification and ranking of transiting exoplanet candidates in Kepler light curves, including the discovery of two new exoplanets] and time series [George and Huerta, 2018b,a, identification of gravitational wave signatures within noisy time series data -a solution that scales better than template matching as the number of templates grows ]. DBSCAN [Castro-Ginard et al., 2018] and k- NN [Smirnov and Markov, 2017] have been used to find structures in multi-dimensional catalogues. Given the prevalence of imaging data in astronomy, it is not surprising that images are being analyzed with the largest range of ML/AI methods.
There is still plenty of scope for studies that perform structured comparisons between multiple methods. This can occur more easily when reference datasets are made accessible to the community. For example, the availablility of the PHoto-z Accuracy Testing datasets [Hildebrandt et al., 2010] allowed Cavuoti et al. [2012] and Brescia et al. [2013] to establish the efficacy of a multi-layer perceptron (i.e. neural network) method coupled with the Quasi Newton Algorithm (MLPQNA) at assigning photometric redshifts for galaxies and quasars respecitvely. Training on the PHAT-1 spectroscopic sample, MLPQNA out-performed alternative statistical and neural network-based methods [e.g. ANNz; Collister and Lahav [2004]] with regards to bias 10 for all objects, bright objects, and distant vs near objects in the PHAT-1 sample [Cavuoti et al., 2012]. Studies into the accuracy and validity of ML-based photometric redshifts continues today -see, for example, Almosallam et al. [2016], Cavuoti et al. [2017a] and Amaro et al. [2019].
Looking more broadly, these systematic comparisons tend to be occurring more often in solar astronomy than in other disciplines [e.g. Nishizuka et al. [2017], Florios et al. [2018] and Inceoglu et al. [2018]], although see Ksoll et al. [2018], Pashchenko et al. [2018, and  for examples pertaining to stellar and variable star classifications. While certain disciplines have adopted specific methods, experimentation with emerging techniques is on-going [e.g. probabilistic random forests and transfer learning Reis et al. [2019]].

Assessing the maturity of adoption
The seven categories introduced in Section 2.2 allow an assessment of the maturity of the use of machine learning and artificial intelligence within a sub-field of astronomy, as they represent a loose hierarchy of sophistication. The common starting point is to apply a machine learning technique to perform a classification, regression or clustering task. Once established as being comparable to, or exceeding, a more traditional approach, machine learning can be used to forecast likely future outcomes [e.g. solar flares [Nishizuka et al., 2017, Florios et al., 2018 or coronal mass ejections from the Sun [Inceoglu et al., 2018]] or make new discoveries [e.g. classification schemes for stellar types permitting the identification of new candidates of rare objects as in Bu et al. [2017], van Roestel et al. [2018, and ]. The most mature disciplines move beyond classification and discovery as ends in their own to that of gaining insight -where new physical knowledge is identified, often for the first time, because a machine learning approach was used.
The hierarchy of categories is used to assess the maturity of ML and AI within a set of sub-fields of astronomy as one of emerging, progressing, or established. In all cases, the reader should refer to the highlighted works in order to understand the scientific background, historical context for the establishment of a particular method, and the technical details of the data mining, machine learning or artificial intelligence approach that was applied.

Emerging
The emerging stage is applied to sub-fields of astronomy and astrophysics that are starting to investigate the use of ML and AI, often by tackling the "low-hanging fruit". This includes a problem that requires a classification or regression approach, or through a comparison between machine learning and an alternative, established method. While some of the emerging disciplines show evidence of reaching the discovery and insight phases, the approaches are not as firmly established, or the size of the community is small. Emerging fields include: • Planetary studies. ML-based identification and classification of clouds, dust storms and surface features on Mars [Gichu and Ogohara, 2019], with potential to forecast future dust storms [Mukkavilli et al., 2018], and the discovery of previously unknown impact craters [Xin et al., 2017] using the AdaBoost algorithm.
• Non-stellar components of the Milky Way. The primary components of the Milky Way are stars (see below), dust and gas, which can be concentrated in atomic and molecular clouds or more diffusely in the interstellar medium. Ucci et al. [2018] developed the GAME (GAlaxy Machine learning for Emission lines) code to study physical properties of the interstellar medium. Segmentation and clustering algorithms are used to identify individual components of clouds of atomic and molecular gas within the Milky Way. Bron et al. [2018] used the Meanshift clustering algorithm to identify regions within molecular clouds that based on physical/chemical properties, instead of seeking purely spatial connections; Dénes et al. [2018] used ML to determine individual Guassian components of the Riegel-Crutcher cloud, based on 21-cm neutral hydrogen (Hi) observations of extra-galactic continuum sources behind the cloud complex; and SVM was used by  to classify, and hence select, Hii regions (gas clouds comprised mostly of singly-ionized hydrogen) from the Infrared Astronomical Satellite (IRAS) Point Source Catalogue. Time-consuming human classifications of Milky Way "bubbles', caused by stellar feedback within molecular clouds, was enhanced through ML [Xu and Offner, 2017]. A random forest method was used by Chen et al. [2019] to determine the amount of dust reddening of more than 56 million stars, with application to future GAIA datasets. Using a GAN, Vavilova et al. [2018] demonstrated how to generate missing parts of the cosmological large-scale structure that are obscured by the Milky Way's zone of avoidance.
• Stellar clusters. Application of the density-based DBSCAN algorithm to the Gaia Data Release 2 (DR2) has lead to the discovery of new open clusters [Castro-Ginard et al., 2018], with an ANN used to separate real clusters from spatial over-densities of stars. Multi-dimensional clustering processes have been used to determine the components of several open clusters, including M67 Gao [2018a] and NGC188 [Gao, 2018b].
• Instrumentation.  presented an AI solution for identifying faults for a telescope drive system. Through an automated expert system, a series of self-healing decisions are made until an appropriate solution is found. The knowledge base is updated in real-time with human input for faults that have not previously been diagnosed or corrected.
Other emerging disciplines include: information retrieval systems, matching queries regarding specific instruments [Mukund et al., 2018]; identification of cosmic strings in all-sky maps [e.g. Ciuca and Hernández [2017] using a neural network; Vafaei Sadr et al. [2018]]; and the detection and classification of cosmic ray events [Krause et al., 2017.

Progressing
Characteristics of disciplines identified as progressing in their use of ML and AI include a broader variety of techniques being applied, or a particular technique is used multiple times, or there is an immediate move to the forecasting, discovery or insight phases. Sub-fields at this stage of maturity include: • Solar System objects. Due to their relative proximity to the Earth, the motion of Solar System objects is key to their discovery. ML and AI have enabled removal, or reduction, of false detections from the moving object detection pipeline in the Subaru/Hyper-Suprime-Cam Strategic Survey Program , with applications to Trans-Neptunian Objects ; and detection and classification of asteroids [Erasmus et al., 2017, Smirnov and Markov, 2017. Duev et al.
[2019] trained a CNN to discover fast-moving candidates from ZTF observations in order to more reliably identify potentially hazardous near-Earth objects.
• Active galactic nuclei and quasars. A common theme in this field is the need for classification and detection methods, including assigning morphological types to radio-detected active galactic nuclei with a CNN , identifying blazar candidates in the Fermi-LAT (3LAC) Clean Sample [Kang et al., 2019], detecting rare high-redshift, extremly luminous quasars [Schindler et al., 2017], and discriminating populations of broad absorption line quasars (BALQs) from non-BALQs in SDSS data releases [Yong et al., 2018].
• Cosmological simulations. ML is providing new methods for examining the outputs of cosmological simulations, leading to new insights about the connections between physical properties of galxies, dark matter halos and the cosmic environment. Examples include the use of an ANN to aid in determining the total mass of the Milky Way and the Andromeda Galaxy from the Small MultiDark simulation [McLeod et al., 2017], and both classification of sub-halos [Nadler et al., 2018] and assignment of galaxies to halos [Agarwal et al., 2018] in dark matter-only simulations.

Established
In the established sub-fields, the use of ML and AI have become essential, a substantial body of literature exists, and the focus is mostly on forecasting, discoveries, or insight. Here, there is no longer a need to evaluate a suitability of machine learning -its usage has become ingrained. Established sub-fields include: • Solar astronomy. Machine learning has been used for classification of solar flares [e.g., Liu [2017], , and  via a hybrid method using both supervised and unsupervised methods]; clustering [e.g.  presented the simulated annealing genetic (SAG) AI method to distinguish between the umbra, penumbra and solar photosphere through a segmentation approach]; and forecasting of coronal mass ejections with a SVM , and SVM and multilayer perceptrons [Inceoglu et al., 2018]. A number of systematic comparison studies have been conducted in order to assess which ML methods perform best at forecasting solar events. Nishizuka et al. [2017] compared three ML methods to forecast solar activity from time-based features using data from the SDO and the Geostationary Operational Environmental Satellite (GEOS) system. They determined that k-NN performed more effectively than SVM or extremely randomized trees. For a different data set from the SDO, Florios et al. [2018] found that random forests provided greater prediction accuracy.
• Extra-solar planets. The Kepler Mission, amongst other search programs, has been a rich source of light curve data. ML and AI have enhanced and improved candidate selection [Armstrong et al., 2018], and classification of light-curves, by removing false positives [Armstrong et al., 2017], raising detection efficiency [Mislis et al., 2018] and accuracy [Pearson et al., 2018], and identifying anomalies [Giles and Walkowicz, 2019]. ML techniques have allowed discovery of fainter candidates than were accessible with existing methods using neural network and random forest algorithms [Gomez Gonzalez et al., 2018] and a CNN [Shallue and Vanderburg, 2018]. Insight into candidate and confirmed extrasolar planets is also being achieved with ML and AI, such as through the determination of a "habitability score" for extra-solar planets  and improved model-fitting of atmospheric composition [Márquez-Neila et al., 2018].
• Stars and stellar products. Two key activities in stellar astronomy are spectral classification [e.g. ; Garcia-Dias et al. [2018] with k-means clustering; Kong et al. [2018]; classification of young stellar objects with eight different methods by Miettinen [2018]] and photometric classification [e.g., Ksoll et al. [2018];  with SVM, RF and Fast Boxes].
Many new examples of specific stellar classes have been discovered, such as Wolf-Rayet stars [Morello et al., 2018], blue horizontal branch stars [Wan et al., 2018], hot sub dwarf stars [Bu et al., 2017], and rare hypervelocity stars [Marchetti et al., 2017]. ML/AI have also led to the discovery of unresolved binary stars in simulated catalogues using RF and ANN algorithms [Kuntzer and Courbin, 2017], and new pulsars, and fewer false postives, from the LOFAR Tied-Array All-Sky Survey . Fujimoto et al. [2018] used an ANN to gain new insights into neutron star equation of state from numerical simulations.
• Variable stars. Time-based photometric observations of variable stars provide extensive data sets of many millions of individual objects for mining. There has been highly productive use of ML and AI for classification [e.g. Benavente et al. [2017], Naul et al. [2018], and Papageorgiou et al. [2018]] and discovery activities [e.g. comparative study with a variety of techniques, including SVM, k-NN, neural networks and random forests by Pashchenko et al. [2018]; discovery of new EL CVn-type binaries from the Palomar Transient Factory [van Roestel et al., 2018] supported with data from other large-scale surveys]. Valenzuela and Pichara [2018] used an unsupervised method to overcome the limitations of available training samples, particularly when approaching new survey data, by identifying similariaties between light curves rather then relying on pre-classified examples.
While not yet the preferred method for all observational programs, the applicability of ML and AI has been firmly established -particularly for the real-time detection of transient objects, which allows new classes of celestial objects to be discovered. The Catalina Real-Time Transient Survey ] was utilized as a large-scale test-bed for the use of ML to aid the optical detection and monitoring of variable and transient objects [e.g. , Djorgovski, 2014, Djorgovski et al., 2016. These early successes paved the way for the Zwicky Transient Factory . The multi-filter optical survey with the ZTF was designed to provide a rich exploration of the transient sky, generating hundreds of thousands of real-time candidate alerts for each night of operation. Machine learning is fundamental to accelerating and enabling the data analysis and candidate identification (and rejection) workflows of the ZTF. ML is being used to perform time critical tasks such as morphological star/galaxy classification [Tachibana and Miller, 2018], binary real/bogus classification of candidates, and asteroid detection , and can play a role in the brokering of alerts with application to the LSST Alert Stream [Narayan et al., 2018]. In a radio-based transient object project, Farah et al. [2018] used a random forest as part of the UTMOST real-time detection pipeline, leading to the discovery of Fast Radio Burst FRB170827, however, Connor and van Leeuwen [2018] determined that CNNs were sub-optimal for some radio transient tasks, such as reducing the need for GPU-acclerated, brute-force dedispersion of time series signals.
• Galaxies. One of the major areas of ML application has been in the classification of galaxies from optical and radio imaging surveys. Recent examples include: neural network-based Faranoff-Riley classifications of radio galaxies [Aniyan and Thorat, 2017]; automated morphological annotation and assignment [Beck et al., 2018, Kuminski and Shamir, 2018 and labelling [Hocking et al., 2018] of galaxy images, including detections from radio surveys [Lukic et al., 2018]. The reference point for many of the automated classifiers is the work done by human volunteers for projects like Galaxy Zoo [Lintott et al., 2008].  uncovered human biases that existed in morphological classification, which could be reduced through supervised ML. Other applications included predicting the Hi content of galaxies based on optical observations [Rafieferantsoa et al., 2018], determining physical properties of galaxies from their emission-line spectra [Ucci et al., 2017], point source detection from radio interferometry surveys [Vafaei Sadr et al., 2019], and cross-identification of sources from the Radio Galaxy Zoo [Alger et al., 2018]. Training a CNN on mock images of rare "blue nugget" galaxies from cosmological simulations, such objects were successfully found in an observational sample from the CANDELS survey .
• Distance measures. Estimates of the distances to galaxies, quasars and other remote celestial objects has benefited greatly from the adoption of ML. Redshifts can be accurately inferred from photometric measurements of galaxies [e.g. Koo, 1985, 1999, Bolzonella et al., 2000, by training on samples where spectroscopic redshifts are also available. In general, it is more challenging to make the required spectroscopic measurements for large samples of galaxies [Ball and Brunner, 2010], whereas many surveys are able to provide a wealth of features for ML algorithms to learn from. Recent work has included: comparisons between Gaussian Processes and other machine learning methods -including ANNz [Collister and Lahav, 2004] -from SDSS Data Release 12 Almosallam et al. [2016]; removal of anomalies from training data Hoyle et al. [2015]; application of deep neural networks  and SVM Jones and Singal [2017]; and the use of k-means clustering to identify features for input to photometric redshift estimation from SDSS datasets [Stensbo-Smidt et al., 2017]. Multiple machine learning methods have been utilized for determining photometric redshifts for the Dark Energy Survey Science Verification shear catalogue (DES SV) Bonnett et al. [2016]. See also Morrison et al. [2017], Cavuoti et al. [2017a], Leistedt and Hogg [2017] Jacobs et al. [2019b,a]. A strong lens finding challenge was recently conducted using simulated data [Metcalf et al., 2019], and deep learning-based methods outperformed all other methodologies including examination by human experts.
• Gravitational wave astronomy The recent detection of gravitational wave signals from coalescing black hole binaries [Abbott et al., 2018], and other related compact systems, has relied on real-time computation and analysis of streams of data from the Advanced Laser Interferometer Gravitational-Wave Observatory (LIGO) detectors [Harry and LIGO Scientific Collaboration, 2010]. By incorporating machine learning, Powell et al. [2017] improved performance in distinguishing between sources and noise signals, along with reducing the latency of the detection pipeline. Zevin et al. [2017] used crowd-sourced categorization of common "glitch" signals in order to train a ML system for real-time glitch classification. George and Huerta [2018b,a] developed Deep Filtering, which utilizes two CNNs for detecting signals (classification) and performing parameter estimation (regression) in real time.
Testing first on mock data, they successfully recovered events from LIGO observations. Theoretical insight into binary black hole mergers has also been achieved through ML, training on outputs from numerical relativity simulations [Huerta et al., 2018].

Concluding remarks
Every week, new astronomical applications of machine learning and artificial intelligence are added to a growing corpus of work. Random forests, support vector machines, neural networks (artificial, deep, and convolutional), and generative adversarial networks are now having a genuine impact across all domains of astronomy. ML and AI simplify the processes of classification and regression, determination of clustering relationships, forecasting of time-based events, and the generation or reconstruction of missing information.
As methods become more sophisticated, the volume of training data grows, and classifications become more robust, ML and AI allow for new objects to be discovered, and for new scientific insight to be gathered. The adoption of ML and AI is emerging in planetary studies, investigations of the non-stellar components of the Milky Way and of stellar clusters, and in real-time monitoring of instruments. Elsewhere, rapid progress is occurring in the use of ML and AI for classification and detection of Solar System objects, and the discovery of rare types of active galactic nuclei and quasars. ML, in particular through early experimentation with GANs, offer an intriguing alternative to generating and understanding complex structures in cosmological simulations. ML and AI are now firmly established in: solar astronomy (particularly forecasting of solar activity); the discovery of extra-solar planets and transient objects; and classification, discovery and gaining insights into the properties of all types of stars, variable stars, and stellar evolutionary products (neutron stars, pulsars, and black holes). ML and AI offer new ways to find and understand galaxies, gravitationally-lensed sources, and gravitational wave candidates.
As astronomy moves ever closer to the Exascale data era of the Square Kilometre Array, an increasing number of human-centred tasks and processes are being replaced by faster, automated processing. The adoption of ML and AI techniques is driving a fundamental change in the way future astronomers will approach the process of "discovery". To date, in the vast majority of cases, discoveries have occurred when astronomers look directly at their data: qualitative inspection supported by quantitative analysis (e.g. model fitting, simulation, etc.). The volume (e.g. number of sources or quantity of data recorded per source) and the complexity (e.g. dimensionality) of data has not vastly exceeded available computing or visual resources. It has been possible to look at the majority of potential sources and false detections by eye, and to conduct visualization and analysis using a desktop-bound workspace. This is no longer the case. Continuous human monitoring of data streams from the SKA is likely to be a tedious task, and discoveries will be missed. Computers excel at such repetitive actions [see, for example, Yeakel et al. [2018], who investigated the role of AI as a means to reduce tedium and detect anomalies in spacecraft systems, through the Cassini-Huygens mission's study of Saturn's magnetosphere], and allow astronomers to focus their attention on interpreting and explaining new types of astronomical phenomena and their connection to fundamental physics.
As the use of deep neural networks increases in astronomy (and a great many other fields), the question arises: what is going on inside the networks? Where AI systems are making medical diagnoses and driving autonomous vehicles this may be an urgent question; but an understanding of the errors, biases and limitations is also of growing importance in a scientific context. In the computer vision realm several attempts have been made to develop techniques for visualizing and interpreting deep neural network outputs [see Montavon et al., 2018], for instance visualizing the feature detectors or building "saliency maps" of the important input pixels [Selvaraju et al., 2017]. However, the utility and adequacy of these methods in astronomy, where precisely quantified errors are often required, is far from apparent. Several advancements have been made, for example the use of Bayesian neural networks [Denker and Lecun, 1991] -where the outputs are probability distributions -in estimating gravitational lens model errors [Perreault Levasseur et al., 2017] or the uncertainties in neutron capture mass models [Utama and Piekarewicz, 2017]. Further work in understanding the internals of deep networks will be needed if the promise of deep neural networks for astronomical discovery is to be fully realized.
As progress in artificial intelligence and machine learning accelerates, particularly through advances in deep learning, the gap between human and automated pattern recognition capabilities is narrowing. However, it is still not always obvious why and how classifications or discoveries are made by ever more complex neural networks. There is still scope for more studies that consider the strengths and weaknesses of different ML and AI approaches when applied to a specific dataset -particularly as new, experimental techniques continue to appear. Learning which types of objects are harder to detect or classify also provides insight, along with highlighting potential biases in human input. As Ball and Brunner [2010] stated, and as still holds true, "there is no simple method to select the optimal algorithm to use". For the time being, traditional statistical methods or visualization are still highly productive first steps, providing astronomers with a detailed understanding of their data. In the future, there is no doubt that the reach and maturity of machine learning and artificial intelligence in astronomy will continue to grow. acknowledgements CJF and CJ thank WIREs for the invitation and opportunity to write this advanced review, and the two anonymous referees for their insightful comments. CJF thanks Wael Farah for helpful discussions. This research has made extensive use of NASAs Astrophysics Data System.


Detecting GAN generated Fake Images using Co-occurrence Matrices

The advent of Generative Adversarial Networks (GANs) has brought about completely novel ways of transforming and manipulating pixels in digital images. GAN based techniques such as Image-to-Image translations, DeepFakes, and other automated methods have become increasingly popular in creating fake images. In this paper, we propose a novel approach to detect GAN generated fake images using a combination of co-occurrence matrices and deep learning. We extract co-occurrence matrices on three color channels in the pixel domain and train a model using a deep convolutional neural network (CNN) framework. Experimental results on two diverse and challenging GAN datasets comprising more than 56,000 images based on unpaired image-to-image translations (cycleGAN [1]) and facial attributes/expressions (StarGAN [2]) show that our approach is promising and achieves more than 99% classification accuracy in both datasets. Further, our approach also generalizes well and achieves good results when trained on one dataset and tested on the other.

Introduction
Recent advances in Machine Learning and Artificial Intelligence have made it tremendously easy to create and synthesize digital manipulations in images and videos. In particular, Generative Adversarial Networks (GANs) [3] have been one of the most promising advancements in image enhancement and manipulation. Due to the success of using GANs for image editing, it is now possible to use a combination of GANs and off-the-shelf image-editing tools to modify digital images to such an extent that it has become difficult to distinguish doctored images from normal ones. The field of digital Image Forensics develops tools and techniques to detect manipulations in digital images such as splicing, resampling and copy move, but the efficacy and robustness of these tools on GAN generated images is yet to be seen. To address this, we propose a novel method to automatically identify GAN generated fake images using techniques that have been inspired from classical steganalysis.
The seminal work on GANs [3] cast the machine learning field of generative modeling as a game theory optimization problem. GANs contain two networks -the first network is a generative network that can generate fake images and the second network is a discirminative network that determines if an image is real or fake. Encoded in the GAN loss function is a min-max game which creates a competition between the generative and discriminative networks. As the discriminative network becomes better at distinguishing between real and fake images, the generative model becomes better at generating fake images.
GANs have been applied to many image processing tasks such as image synthesis, super-resolution and image completion. Inspired by the results of these image processing tasks, GANs have brought in novel attack avenues such as computer generated (CG) faces [4], augmenting faces with CG facial attributes [2], and seamless transfer of texture between images [1], to name a few. Two of the most common applications of GANs include texture or style transfer between images and face manipulations. An example of GAN generated texture translation between images, such as horses-to-zebras and summer-to-winter, is shown in Fig. 1(a). These techniques manipulates the entire image to change the visual appearance of the scene [1]. There has been also tremendous progress in facial manipulations -in particular, automatic generation of facial attributes and expressions. Fig. 1(b) shows one such recent example where various facial attributes such as hair, gender, age and skin color, and expressions such as anger, happiness and fear are generated on faces of celebrities [2].
While the visual results are promising, the GAN based techniques alter the statistics of pixels in the images that they generate. Hence, methods that look for deviations from natural image statistics could be effective in detecting GAN generated fake images. These methods have been well studied in the field of steganalysis which aims to detect the presence of hidden data in digital images. One such method is based on analyzing co-occurrences of pixels by computing a co-occurrence matrix. Traditionally, this method uses hand crafted features computed on the co-occurrence matrix and a machine learning classifier such as support vector machines determines if a message is hidden in the image [5,6]. Other techniques involve calculating image residuals or passing the image through different filters before computing the co-occurrence matrix [7,8,9].
Inspired by steganalysis and natural image statistics, we propose a novel method to identify GAN generated images using a combination of pixel co-occurrence matrices and deep learning. Here we pass the co-occurrence matrices directly through a deep learning framework and allow the network to learn important features of the co-occurrence matrices. We also avoid computation of residuals or passing an image through various filters, but rather compute the co-occurrence matrices on the image pix-(a) Images generated using CycleGAN [1] (b) Images generated using StarGAN [2] Figure 1: Examples of images that have been generated using GANs. els itself. Experimental results on two diverse and challenging datasets generated using GAN based methods show that our approach is promising and generalizes well when the GAN model is not known during training.

Related Work
Since the seminal work on GANs [3], there have been several hundreds of papers on using GANs to generate images. These works focus on generating images of high perceptual quality [10,11,12,13,14,15,4], image-to-image translations [13,16,1], domain transfer [17,18], super-resolution [19], image synthesis and completion [20,21,22], and generation of facial attributes and expressions [23,24,18,2]. Several methods have been proposed in the area of image forensics over the past years [25,26,27,28,29]. Recent approaches have focused on applying deep learning based methods to detect tampered images [30,31,32,33,34,9,35] The detection of GAN images is a new area in image forensics and there are very few papers in this area [36,37,38,39,40,41,42,43,44,45]. Related fields also include detection of computer generated (CG) images [46,47,48,49]. The most relevant work is a recent paper [36] on detecting GAN based image-toimage translation generated using cycleGAN [1]. Here the authors compare various existing methods to identify cycleGAN images from normal ones. The top results they obtained using a combination of residual features [50,9] and deep learning [51]. Similar to [36], the authors in [38] compute the residuals of high pass filtered images and then extract co-occurrence matrices on these residuals, which are then concatenated to form a feature vector that can distinguish real from fake GAN images. In contrast to these approaches, our approach does not need any image resid-uals to be computed. Rather, our method directly computes cooccurrence matrices on the three color channels which are then passed through a deep convolutional neural network (CNN) to learn a model that can detect fake GAN generated images.

Methodology
To detect GAN images, we compute co-occurrence matrices on the RGB channels of an image. Co-occurrence matrices have been previously used in steganalysis to identify images that have data hidden in them [5,6,7,8] and in image forensics to detect or localize tampered images [50,9]. In prior works, cooccurrence matrices are usually computed on image residuals by passing the image through many filters and then obtaining the difference. Sometimes features or statistics are also computed on the matrices and then a classifier is trained on these features to classify data hidden or tampered images [5].
However, in this paper, we compute co-occurrence matrices directly on the image pixels on each of the red, green and blue channels and pass them through a convolutional neural network, thereby allowing the network to learn important features from the co-occurrence matrices. An overview of our approach is shown in Fig. 2. Specifically, the first step is to compute the co-occurrence matrices on the RGB channels to obtain a 3x256x256 tensor. This tensor is then passed through a multi-layer deep convolutional neural network: conv layer with 32 3x3 convs + ReLu layer + conv layer with 32 5x5 convs + max pooling layer + conv layer with 64 3x3 convs + ReLu layer + conv layer with 64 5x5 convs + max pooling layer + conv layer with 128 3x3 convs + ReLu layer + conv layer with 128 5x5 convs + max pooling layer + 256 dense layer + 256 dense layer + sigmoid layer. A variant of adaptive

Experiments
We evaluate our method on two diverse and challenging datasets which contain GAN generated images such as image-toimage translation, style transfer, facial attributes and expressions.

Datasets
CycleGAN dataset: This dataset contains unpaired image-toimage translations of various objects and scenes such as horsesto-zebras, summer-to-winter, images to paintings (Monet, Van Gough), style transfer such as labels to facades, and others that were generated using a cycle-consistent GAN framework [1]. We followed the instructions provided by the authors as detailed in https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix, and obtained 36,302 images (18, [52] of celebrity faces and the remaining 17,991 images were GAN generated images with 5 varying facial attributes such as black hair, blond hair, brown hair, gender change, aged and 4 combinations of the same. We followed the instructions in https://github.com/yunjey/stargan to generate the images.

Evaluation
We first evaluate our approach on two datasets separately and then perform cross evaluation on the two datasets (one dataset as training and other as testing) to see the generalizability of our approach. For both datasets, 50% of the data is used for training, 25% for validation and 25% for testing. We train the network for 50 epochs with a batch size of 40 and use a variant of adaptive stochastic gradient as optimizer. Fig. 3(a,b) and Fig. 3(c,d) show the model accuracy and loss on cycleGAN dataset and Star-GAN dataset, respectively. We obtained a high training and validation accuracy of 99.90% and 99.40% on cycleGAN dataset, and 99.43% and 99.39% on StarGAN dataset respectively. We 1 We thank the authors for providing the dataset distribution

Generalizability
Next, we evaluate the generalizability of our approach by training on one dataset and testing on the other. First we train on all the images in the cycleGAN dataset (35,302 images) and test the model on all images of the StarGAN dataset (19,990) images, and then we reverse the experiment where we train on StarGAN and test on cycleGAN. We train the network till 50 epochs and report on the model that gave the highest accuracy. As shown in Tab. 1, our method still maintains a high accuracy even across diverse datasets. The model trained on cycleGAN dataset has a higher accuracy of 99.45% in comparison with the model trained on StarGAN dataset which got 93.42%. The lower accuracy for the model trained on StarGAN dataset could be because of the non-uniform distribution of class samples in the StarGAN dataset, and due to the diverse image sources/categories in the cycleGAN dataset.

Comparison with State-of-the-art
We compare our approach with the results presented in [36].
Here the authors conduct a study on the detection of images manipulated by GAN-based image-to-image translation on the cycleGAN dataset. For evaluation, they adopt a leave-onemanipulation-out strategy on the categories in the cycleGAN dataset, where at each iteration images belonging to one category are set aside for validation and the images from other categories are used for training. The methods evaluated are based on steganalysis, generic image manipulations, detection of computer graphics, a GAN discriminator used in the cycleGAN paper, and generic deep learning architecture pretrained on ImageNet [53], but fine tuned to the cycleGAN dataset. Among these the top preforming ones were from steganalysis [8,50] based on extracting features from high-pass residual images, a deep neural network designed to extract residual features [9] (denoted by Coz-zolino2017) and XceptionNet [51] deep neural network trained   on ImageNet but fine-tuned to this dataset. We report the results as mentioned in the paper only from these three methods and compare with our approach.
Results on original images: Here we consider the images generated from the cycleGAN dataset as it is and do not perform any postprocessing on the images. Tab. 2 summarizes the results of our proposed approach along with with the three top performing approaches in [36]. On average our method outperformed other methods and was able to achieve an accuracy of 97.84. On most categories, our approach was better than or on-par with the other top methods. The only categories where our method performed poorly were 'cityscapes' and 'facades'. These could be because the original images in these categories were JPEG compressed which could have affected the classification accuracy. In the next section, we study the effect of compression on our method.

Effect of JPEG compression:
In [36], the authors investigated the sensitivity of their methods on compression. Using a compression method similar to Twitter, they trained their detection methods on original uncompressed images and tested on JPEG compressed images. The objective of this study was to test the robustness of the detection techniques when images are posted in social networks such as Twitter. In the second scenario, they train and test on the JPEG compressed images. We performed both of these experiments on the cycleGAN dataset. Since we are not aware of the exact JPEG quantization tables used in Twitter, our approach was similar but we tested on three different JPEG quality factors (QF): 95, 85 and 75. We used 50% of the data for training, 25% for validation and 25% for testing. The results are reported on the 25% testing data of the cycleGAN dataset (9,076 images). As shown in Tab. 3, the accuracy progressively drops as the QF decreases from 95-75, when trained on the original im-ages. But when the JPEG compressed images are used for training, the accuracy shows a substantial increase. Even at a QF of 75, the accuracy is still 87.31%. This is also consistent with the results reported in [36], where they report close to a 10% drop in accuracy on Twitter-like compressed images.

Conclusions
In this paper, we proposed a novel method to detect GAN generated fake images using a combination of pixel co-occurrence matrices and deep learning. Co-occurrence matrices are computed on the color channels of an image and then trained using a deep convolutional neural network to distinguish GAN generated fake images from real ones. Experiments on two diverse GAN datasets show that our approach is both effective and generalizable. In future, we will consider localizing the manipulated pixels in GAN generated fake images. . Amit K. Roy-Chowdhury received the Bachelors degree in Electrical Engineering from Jadavpur University, Calcutta, India, the Masters degree in Systems Science and Automation from the Indian Institute of Science, Bangalore, India, and the Ph.D. degree in Electrical and Computer Engineering from the University of Maryland, College Park. He is a Professor of Electrical and Computer Engineering and a Cooperating Faculty in the Department of Computer Science and Engineering, University of California, Riverside. His broad research interests include computer vision, image processing, and vision-based statistical learning, with applications in cyber-physical, autonomous and intelligent systems. He is a coauthor of two books: Camera Networks: The Acquisition and Analysis of Videos over Wide Areas, and Recognition of Humans and Their Activities Using Video. He is the editor of the book Distributed Video Sensor Networks. He has been on the organizing and program committees of multiple computer vision and image processing conferences and is serving on the editorial boards of multiple journals. He is a Fellow of the IEEE and IAPR.


extfieldsofstudy:
Mathematics
Computer Science

text:
The Application of Bipartite Matching in Assignment Problem

The optimized assignment of staff is of great significance for improving the production efficiency of the society. For specific tasks, the key to optimizing staffing is personnel scheduling. The assignment problem is classical in the personnel scheduling. In this paper, we abstract it as an optimal matching model of a bipartite graph and propose the Ultimate Hungarian Algorithm(UHA). By introducing feasible labels, iteratively searching for the augmenting path to get the optimal match(maximum-weight matching). And we compare the algorithm with the traditional brute force method, then conclude that our algorithm has lower time complexity and can solve the problems of maximum-weight matching more effectively.

INTRODUCTION
M OTIVATED by the increasing in labor costs, employing multi-skilled workers has become an important way to raise the personnel utilization. For a specific task, personnel scheduling is the key to optimizing staffing. Assignment problem is one of the classic problem in the staff scheduling. m workers match n tasks, and each worker has different efficiency in each task, determine a task allocation plan which makes the highest gross efficiency. Traditional approach to this problem is manually allocate the tasks according to the leader's understanding of the workers, which is time-consuming, laborious, inefficient and relying on experience. However, this approach can not meet the requirements of quantitative, rapid and automatic scientific management. Therefore, it is necessary to introduce the theories of operational research to solve the assignment problem with computer assistance.
In this paper, we abstract the maximum-weight matching model from assignment problem, and propose UHS to solve it. The main idea of UHS is to figure out the maximumweight matching by searching augmenting path. Compared to the traditional method, our new method liberates the human labor and find maximum-weight matching more precisely.

Assumption
When dealing with assignment problem, we use subsets of vertices to stand for workers and tasks, and edges between two vertex from different subsets means the worker and task can be matched. We assume that each worker's ability for tasks is quantifiable, which reflects in the weights of edges. Meanwhile, each worker can match up with only one task in our assumption.

Bipartite graph
Given a unidirectional graph G = (V, E), if the vertex set can be partitioned into V = P ∪ Q, P and Q are disjoint and all edges in E go between P and Q, then G called a Bipartite graph.

Matching
A Matching is a subset of edges M ⊆ E such that for all vertices v ∈ V, at most one edge of M is incident on v. A maximum matching is a matching M for any matching M , we have |M | ⊆ |M|. And maximum-weight matching is the matching of the largest sum of weights.

Augmenting path
A path is augmenting for a matching M if it alternates between edges in the matching and edges not in the matching, and the first and last vertices are unmatched.

Feasible Labeling
The majority of realistic matching problems are much more complex. This added complexity often stems from graph labeling, where edges or vertices labeled with quantitative attributes, such as weights, costs, preferences or any other specifications, which adds constraints to potential matches.
A common characteristic investigated within a labeled graph is a known as feasible labeling, where the label, or weight assigned to an edge, never surpasses in value to the addition of respective vertices' weights. This property can be thought of as the triangle inequality.
Where X is the set of nodes on one side of the bipartite graph, Y is the other set of nodes, l(x) is the label of x, etc., and w(x, y) is the weight of the edge between x and y.

Proof
Lemma: A feasible labeling on a perfect match returns a maximum-weighted matching.
Suppose each edge e in the graph G connects two vertices, and every vertex is covered exactly once. With this, we have the following inequality: Where M is any perfect matching in G created by a random assignment of vertices, and l(x) is a numeric label to node x. This means that is an upper bound on the cost of any perfect matching. Now let M be a perfect match in G, then So w(M ) ≤ w(M) and M is optimal.

Implementation
The algorithm starts by labeling all nodes on one side of the graph with the maximum weight. This can be done by finding the maximum-weighted edge and labeling the adjacent node with it. Additionally, match the graph with those edges. If a node has two maximum edges, do not connect them.
If the matching is perfect, the algorithm is done as there is a perfect matching of maximum weights. Otherwise, there will be two nodes that are not connected to any other node. If this is the case, begin iterating. Improve the labeling by finding the non-zero label vertex without a match, and try to find the best assignment for it. Formally, the UHA matching algorithm can be executed as defined in the Algorithm 1.
Each step will increase the size of the matching M or it will increase the size of the set of labeled edges, E l . This means that the process will eventually terminate since there are only a limited number of edges in the graph G.
When the process terminates, M will be a perfect matching. By the Kuhn-Munkres theorem, this means that the matching is a maximum-weight matching.

Input:
The labeling l, an equality graph G l = (V, E l ), an initial matching M in G l , and an unmatched vertex v ∈ V and u / ∈ M; Output: Maximum-weighted matching; 1: We will keep track of a candidate augmenting path starting at the vertex u; 2: If the algorithm finds an unmatched vertex v, add on to the existing augmenting path p by adding the u to v segment; 3: Flip the matching by replacing the edges in M with the edges in the augmenting path that are not in M (in other words, the edges in E l − M); 4: S ⊆ X and T ⊆ Y where S and T represent the candidate augmenting alternating path between the matching and the edges not in the matching; 5: Let N l (S) be the neighbors to each node that is in S along edges in E l such that N l (S) = {v|∀u ∈ S : (u, v) ∈ E l }; 6: If N l (S) = T, then we cannot increase the size of the alternating path (and therefore can not further augment), so we need to improve the labeling; 7: Let N l (S) be the neighbors to each node that is in S along edges in E l such that N l (S) = {v|∀u ∈ S : (u, v) ∈ E l }; 8: Let δ l be the minimum of l(u) + l(v) − w(u, v) over all of the u ∈ S and v / ∈ T; 9: Improve the labeling l to l : If r ∈ S, then l (r) = l(r) − δ l ; If r ∈ T, then l (r) = l(r) + δ l ; If r / ∈ S and r / ∈ T, then l (r) = l(r);

CONCLUSION
Due to the inefficiency of the traditional manual method and the brute force method, we propose UHA to efficiently solve the assignment problem. Compared with brute force method, we improve the time complexity from O(N!) to O(N 3 ).
When using this algorithm to solve assignment problem, we start with some Matching M, a valid labeling l, where l is defined as the feasible labelling. Then looking for an augmenting path in M until a best matching is found, if an augmenting path does not exist, improve the labeling and then go back to first step.
However, our method is not suitable for the assignment problems which the tasks have specific orders. In the future, we plan to optimize our algorithm to solve this problem.