A Comprehensive Study of Artificial Intelligence and Machine Learning Approaches in Confronting the Coronavirus (COVID-19) Pandemic

The novel coronavirus disease (COVID-19) has spread over 219 countries of the globe as a pandemic, creating alarming impacts on health care, socioeconomic environments, and international relationships. The principal objective of the study is to provide the current technological aspects of artificial intelligence (AI) and other relevant technologies and their implications for confronting COVID-19 and preventing the pandemic’s dreadful effects. This article presents AI approaches that have significant contributions in the fields of health care, then highlights and categorizes their applications in confronting COVID-19, such as detection and diagnosis, data analysis and treatment procedures, research and drug development, social control and services, and the prediction of outbreaks. The study addresses the link between the technologies and the epidemics as well as the potential impacts of technology in health care with the introduction of machine learning and natural language processing tools. It is expected that this comprehensive study will support researchers in modeling health care systems and drive further studies in advanced technologies. Finally, we propose future directions in research and conclude that persuasive AI strategies, probabilistic models, and supervised learning are required to tackle future pandemic challenges.

economic performance. These technologies possess the competence of providing enhanced digital solutions for tackling the challenges throughout the calamity [14][15][16][17] and mitigating the global health crisis of this epidemic. 1,7,18,19 AI is one of the prospective technologies in health care for better understanding and addressing the COVID-19 crisis. 20,21 For the present pandemic purposes, AI can be defined as ML or DL, natural language processing (NLP), and computer vision applications that catalyze to build effective systems using big data for the global health emergency. These applications include: (a) AI-based clinical data analysis and robotic and remote surgery, 22 (b) research and drug discovery, 23 and (c) automatic health care activities, processes, and management. 24 This article addresses the present status of AI, as well as the potential of AI approaches (including ML, DL, and NLP) in health care. It provides a comprehensive study of these AI approaches for better understanding and describing the COVID-19 pandemic crisis.

Background
In the past few decades, human civilization has faced several worsening pandemics and global public health emergencies. Since 2009, the WHO has declared 6 public health emergencies of international concern owing to global spread of diseases of the 2009 H1N1 (Swine flu) epidemic, 25 the 2014 polio outbreak, 26 the 2014 Ebola virus disease in Western Africa, 27 the 2016 Zika virus epidemic, 28 the ongoing 2018 Kivu Ebola epidemic, 29 and the COVID-19 pandemic. 30 The 2019 novel coronavirus (2019-nCoV) outbreak is the newest among the major epidemics and, thus, the WHO declared a health emergency due to the biggest viral disease outbreaks. The eruption of COVID-19 triggering novelcoronavirus-infected-pneumonia has hampered the lives of 132 million people in the globe, reported on April 4, 2021, according to the WHO. 11,12,30 The 2019-nCoV is now rapidly grown and distressing 219 countries and territories and 2 international conveyances ( Figure 1), according to the Worldometer, updated on April 4, 2021. 11 The WHO officially renamed the disease as COVID-19 on February 11, 2020, and the virus that causes the disease as SARS-CoV-2 and declared a global health emergency. Researchers suggested a significant genomic similarity of 2019-nCoV with the severe acute respiratory syndrome (SARS) virus that appeared as a pandemic in 2002. 31 In dealing with the COVID-19 global pandemic, AI and related technologies have potential to help human societies and health care. In health care, AI and ML tools provide significant opportunities in the clinical areas of diabetes, heart disease, cancer, neurological disease, and infectious disease (see Figure 2). 32 These technologies can be introduced to design efficient systems and to support researchers and physicians in addressing and overcoming challenges of the infectious coronavirus crisis. Various companies have launched a series of AI technologies to support organizations in the fight against COVID-19, such as estimation of outbreak size, coronavirus detection, diagnosis, analysis data and treatment  procedures, research and drug development, and the future prediction of outbreak. AI is not one technology, but rather a group of technologies, 33,34 which can make significant contributions in the field of health care ( Figure 3).

Machine Learning and Deep Learning
The most important form of AI is ML, which has uses for various contexts in health care, such as diagnosing patient attributes, predicting treatment procedures, and precision medicine. 35 The composite forms of ML include DL or neural network models used in clinical data analysis and detection of particular diseases. 36,37 Natural Language Processing The applications of NLP include design, construction, identification, understanding, and classification of clinical documents as well as research papers. The NLP can scrutinize unstructured medical information about patients, organize reports from radiological investigations, transcribe patient interactions, and extract information from unstructured data to enrich structured medical data. 38,39 It is combined with the ML-based knowledge graph to find a connection between the coronavirus and potential drug candidates and to provide outputs. 40,41 Expert Systems Expert systems, involving human experts and knowledge engineers, are widely employed for clinical decision making 37 and electronic medical records providers prepare a set of rules with daily procedures in health care.

Robots and Robotic Process
Robots are used to deliver medical supplies in hospitals and other quarantine areas. AI-capable surgical robots can provide superpower to surgeons for improving their abilities; regular surgical measures employing robotic surgery 22 include gynecologic surgery, prostate surgery, and head and neck surgery, among others. Robotic process automation does not actually involve robots, rather utilizing computer programs to perform repetitive tasks, such as authorization, modernizing medical histories, or billing processes in health care. 33 There are several potential contributions of AI against the coronavirus pandemic, such as early detecting, tracing, and forecasting, diagnosis and projection, treatments and medications, and social management and services, 21 which are categorized in the following subsections.

Understanding and Detecting
To prevent the spread of infectious diseases, viruses should be tracked earlier. Quick detection of virus helps for patient isolation, contact tracing, treatment, and the delivery of emergency alerts to others. AI-based systems can learn to detect virus outbreaks by analyzing news reports, social media posts, and other outbreak-related documents. AI system BlueDot, 42 using ML algorithms, was used to track and detect infectious disease outbreaks in Wuhan, China. Another AI predictive system, ClosedLoop C-19 index, 43 using expert knowledge in health care can identify those people who are at the highest risk of critical complications from COVID-19. AI-based surveillance systems, using facial recognition tools and temperature sensors, can identify people who have fevers and who have coronavirus. Several search engines, social networking sites, and media sites using ML approaches can be used to track flu-related disease outbreaks in real time. 44 But the main challenge here is data safety and security. It is required to establish a standard protocol to permit communication between devices and systems without compromising data integrity.

Diagnosis, Genome Analysis, and Treatment
Quick and correct findings of COVID-19 cases can save lives and prevent further outbreaks. AI technology has a superior impact on faster disease identification and diagnosis. 20 It can extract information from various clinical reports, analyze DNA sequences, and advise on treatment procedures. 45 A DL method 46 and a supervised neural network model (known as COVNet) 47,48 were developed to examine the chest x-ray and computed tomography (CT) scan images of patients and identify COVID-19 cases. 49 The model can accurately distinguish between COVID-19(+) and COVID-19(−) patients. 50 Multiple AI companies have launched AI-powered diagnosis systems that can help physicians diagnose the virus and monitor disease outbreaks. In the United States, XR Health 51 recently announced a virtual reality telehealth support group to assist persons in isolation due to the virus, so that patients with similar diseases can get support from each other and from physicians. Also, DL approaches are applied in SARS-CoV-2 genomic sequencing that can process large and complex genomic datasets. [52][53][54] For instance, Google DeepMind's AI algorithms can help to understand the protein structure of the virus and support the development of treatment procedures for new diseases. 55 The AI systems for diagnosis, analysis, and treatment face logistical difficulties in implementation, because of dataset swing, new populations, and the negative consequences of new algorithms on health outcomes.

Prediction and Forecasting the Outbreak
AI has been employed for keeping track of and predicting how COVID-19 spreads over time and space. 21 Akhtar et al 56 presented a dynamic artificial neural network prototype to forecast the span of the COVID-19 pandemic. This approach was applied for the prediction of the 2015 Zika virus pandemic. HealthMap 57 and BlueDot 42 have been developed using ML algorithms that can efficiently predict the outbreak of the virus. Wang et al 58 proposed a prognosis model for influenza diagnosis, depending on analysis of the real-time Twitter information that can help to prevent future outbreaks. 59 The XGBoost model 60 is another ML-based prediction model that was used to diagnose a patient disease caused by the coronavirus. The model is convenient for analyzing whether a person should be isolated due to the COVID-19. Several studies have been performed to gather training data from the present pandemic and to develop a precise prediction model for the pandemic. 61 Google Flu Trends 62 presented an approach that employs online searching for estimating and predicting real-time flu activity using AI and big social data. This forecasting and prediction enable health officials to make better response plans and manage the pandemic. 63 Lacking sufficient datasets, AI-based forecasting models may not represent true transmission patterns.

Speeding up Research and Drug Discovery
Amazon Web Services (AWS) developed a new search engine, CORD-19, 64 using ML and NLP that can help scientists and researchers search huge volumes of research articles and documents quickly. In California, an AI-based company, C3.ai, 65 recently founded a research consortium aimed at tackling COVID-19 by utilizing AI and related strategies in finding the virus spread, forecasting its evolution, developing new drugs, and fighting future outbreaks. 66 AI and ML can identify and predict viral proteins of 2019-nCoV for drug development, support that can speed up the discovery of medications and vaccines to treat COVID-19. 54,55,67 An AI company in the United Kingdom, BenevolentAI, 68 aimed at discovering drugs and immediate medicines for COVID-19 by employing AI, DL, and other cutting-edge technologies. Even before the COVID-19 outbreak, AI had potential contributions to health care for new drug discovery. 69,70 Generally, AI is still in its initial stages in this area, and the prediction accuracy of ML methodologies, together with more real-world relevance, will continue to develop.

Public Health Management and Services
Several AI-based activities have the potential to contribute to management of the pandemic, such as helping doctors to monitor health crises and handle multiple patients in hospitals, scanning infected people in public spaces using thermal imaging, measuring social distancing and lockdown procedures, and providing health consultation services to raise awareness around the world. Also, AI-based systems and virtual reality technology provide social services for COVID-19 patients and help to educate others. Such systems, therefore, illustrate how society and public administrations can benefit from the use of AI technology. AI-based drones are used for safe and fast delivery of medical supplies in critical zones 71 ; AI-enabled robots are used for cleaning, sterilizing, and delivering food and medicine to avoid human-to-human contact; and ML-based UVD robots using ultraviolet light can disinfect and autonomously kill bacteria and viruses to limit the spread of coronavirus. The major threats against the safety of the people and AI devices as well as their interference with air traffic are needed to be addressed.

Related Datasets
Data is a vital factor in helping public, governmental, and health care sectors combat the COVID-19 pandemic. It is key to remember that not all data is complete and accurate. Thus, it is necessary to handle uncertainty and inconsistency in coronavirus datasets. Very recently, different datasets have been used by researchers and scientists for their research related to COVID-19. A few of the used datasets related to the coronavirus pandemic are listed below.

Machine Learning Models for Detection and Diagnosis
ML architectures, such as convolutional neural networks (CNNs), provide widespread applications in medical image analysis for feature selection, classification, segmentation, and lesion detection for medical data. [97][98][99][100] Medical imaging and signal data, such as x-ray, CT, and magnetic resonance imaging, are analyzed with the help of DL models. These analysis results have a great impact on the detection, identification, and diagnosis of several diseases, such as cancer, brain tumor, diabetes, and pneumonia. 50,[101][102][103][104][105] Recently, the reverse transcription-polymerase chain reaction is a standard test method used in detecting and categorizing respiratory viruses. 106 Although this standard method of diagnosis is capable of detecting COVID-19, computeraided ML techniques nevertheless aid faster detection and diagnosis. Accordingly, rapid and cost-effective solutions for fast COVID-19 detection utilize AI-and ML-based techniques 107,108 for clinical image processing on the coronavirus. [109][110][111][112][113] In most of the research, x-ray and CT scan images are used as the input of ML and DL prototypes for early detection of virus-infected cases. 49,50,114,115 Different pretrained DL models, such as ResNet50, InceptionV3, VGG-19, and Inception ResNetV2, have been presented for detecting COVID-19-infected patients by employing chest x-ray and CT scan images. 106,112,115 For the experiment, COVID-19 patients' imaging data have been collected from the GitHub 78 and the Kaggle 87 public repositories. The residual neural network (ResNet) architecture is an improved form of the CNN model 116 and ResNet50 is a 50-layer neural network-based model employing a scheme of skip links between layers, known as residual learning and trained on the ImageNet dataset. 89 InceptionV3, 117 one kind of CNN model that contains a fully connected neural network, improves the use of manipulated resources inside the network. The Xception 118 CNN model developed by Google was an updated form of the Inception model. VGG-19, 119 having 19 layers of deep neural network architectures with low convolution filters, was modeled to reach high accuracy in health care applications with a high volume of image data. 120 Another pretrained architecture contains a deep neural network employing the Inception ResNetV2 model that was trained on the patients' image dataset and provided the output as a list of estimated class probabilities of the diseases. 121 The schematic illustration of conventional CNN, including pretrained ResNet50, InceptionV3, VGG-19, and Inception ResNetV2 architectures for COVID-19 detection and diagnosis, is shown in Figure 4. It provides the 3 types of detection, such as COVID-19(+), COVID-19(−), and others. This model helps health staff determine which cases should be investigated with standard methods and which management approaches should be employed in distinguishing COVID-19 and non-COVID-19 cases.
Narin et al 112 presented 5 pretrained DL models, such as ResNet50, ResNet101, ResNet152, InceptionV3, and Inception-ResNetV2, for finding COVID-19 patients using chest x-ray images. The datasets contain the chest x-ray images of bacterial and viral pneumonia, comprising coronavirus and other cases as well as data from healthy people, using data acquired from the GitHub 78 125 a deep neural network model was implemented that offered the precise diagnostics for multiclustering, such as COVID-19(+), pneumonia, and none, using chest x-ray images. This approach applied the combined ResNet50-support vector machine (SVM) models and performed the classification with 95.38% accuracy. Although ML-based imaging approaches show a significant role in the analysis of COVID-19 cases, the irregularities in datasets remain the prime challenge in COVID-19 diagnosis from image data.

Machine Learning and Natural Language Processing Model for Clinical Decision Making and Management
Data collected from various sources are not always structured and require NLP procedures to extract data features from disorganized data sources to enrich structured medical data. 38 ML-based approaches are then applied to analyze structured clinical data, provide the clustering of patients' reports, and subsequently assess the probability of disease outcomes. 39 Thus, NLP procedures generate machine-readable structured and clustered data from texts, and then the clustered data can be analyzed by ML techniques to provide outputs, as shown in Figure 7. 126 The figure illustrates the process flow of structured data generation from clinical documents, including images, genetic data, electronic medical records, and electrophysiological data, through NLP and ML, to suggest clinical activities.
AI-based models can be applied to solve clinical problems and to assist medical practices in primary detection, identification, diagnosis, treatment, and management of critical COVID-19 cases. [127][128][129] Another AI-based model is shown in Figure 8. DL (or deep neural network) approaches could be useful tools for the differential diagnosis of COVID-19, and the genomic variants from normal or severe COVID-19 patients can be classified by the ML analyzer to predict potential COVID-19-infected patients. 53 This model could also be used to monitor critically ill patients and assist physicians in decision making about further therapy. Although ML models provide significant improvements in clinical decision making, a large volume of studies is needed to ensure these outcomes and to increase their acceptance in health care sectors.

Artificial Intelligence System for Identifying COVID-19 Symptoms
AI technology can be conveniently employed for keeping track of the spread of virus outbreaks, identifying extremely critical patients and regions, and controlling virus infections expeditiously. AI can forecast the death risk and recovery cases by analyzing patients' earlier data. Figure 9 illustrates the methodological steps of an AI-based system that can help medical staff to identify the COVID-19 symptoms and suggest the necessary treatments for COVID-19 patients. 130 The system starts with COVID-19 symptoms and patients' test analysis. After confirming COVID-19-infected cases with the aid of AI, doctors decide further therapy and start treatment procedures and monitoring. The doctors are paying attention to treatment of the patients as well as AI applications to administer the spread of the infection. Thus, AI is a useful tool to identify early infections of COVID-19 and to improve treatment consistency by continuously monitoring infected patients using ML algorithms.

Machine Learning-based Forecasting Methods
Fong et al 63 presented an ML methodology with a small dataset and conducted an investigation with the recent coronavirus eruption found in Hubei, China. It was an optimized forecasting model with a polynomial neural network that was able to forecast with a relatively low prediction error. The prediction architecture was designed with 5 classes of data analytics, and these classes embraced different types of ML algorithms. 131 • Class 01: Prediction employing ML architectures with manifold regression analysis. These models include some extent of neural network architectures that possess extremely configurable organizations and parameters, applied to multivariate datasets. 132 With the perspectives of initial stages of epidemic prediction where long dimensional data are not obtainable, these models would be required for reducing forecasting errors. • Class 02: Prediction employing complex ML architectures. These models employ comparatively complicated ML processes, such as SVM or neural networks, which involve iterative hyperplanes or weights modifications for minimizing errors. Hence, these complex models consume significant time and space complexity. • Class 03: Prediction employing lightweight ML architectures. They are more reliable ML models, which run faster and produce results as conventional time series predictions. • Class 04: Simple data analytics. Most are traditional statistics, such as mean, median, minimum, maximum, increment, rate of increases of suspected cases, rate of recovery, etc. • Class 05: Econometrics kind of time series prediction.
Classical econometrics, such as autoregression, exponential regression, moving average, autoregressive integrated moving average and its variants, etc. These approaches typically are employed on univariate time series with nil or small parameters.
From the study, it is noted that traditional prediction from Class 05 can be initially used, possibly simultaneously with Class 04 statistical analysis. With available resources and enough time, the strategies from Classes 03, 02, and 01 can be used in forecasting models that provide relatively low prediction errors. Moreover, forecasting is a dynamic process whose estimated outcome is sensitive to the model parameters and the availability of data. Thus, it is crucial to test popular ML algorithms with complete datasets in the new field.

Machine Learning for Accelerating Drug Development
Because there are still no explicit medications available for COVID-19, it is crucial to determine quick and effective solutions to inhibit COVID-19 infection and virus incidence. Magar et al 133 introduced an AI-based model that combines big data and medical knowledge with ML to determine the antibody sequences that can prevent the progress of coronavirus. 106 A pictorial diagram of the proposed model is shown in Figure 10, 133 which consists of 4 major modules: training dataset, extracting features, learning, and selecting hypothetical candidates. This ML model was trained on the dataset consisting of 1831 antigen and antibody sequences of numerous viruses (such as H1N1, Dengue, SARS, Ebola, and HIV) from the CATNAP tool 91 and 102 samples of the RCSP protein data bank 92 ; hence, the final dataset consists of 1933 samples. An ML-based cheminformatics software, RDKit, 134 was used to represent the molecular graph and extract the features from the dataset of 1933 samples. Five ML frameworks, including random forest, XGBoost, multilayer perceptron, SVMs, and logistic regression, 133 have been applied to assess their performance, and the experimental results showed that the XGBoost architecture achieved the best performance to determine the impending antibody candidates for COVID-19. Because the COVID-19 virus has similar characteristics as SARS-CoV-2, 135 the researchers recommended producing 2589 hypothetical candidates based on neutralizing antibodies  and sequence features against SARS, and among these candidates, eventually, 8 structures were taken as prospective antibody sequences for neutralizing COVID-19.
As COVID-19 is caused by a novel coronavirus, proper understanding of viral structures helps to accelerate drug discovery. The vaccine development for a new disease is expensive and time-consuming; ML approaches can help to accelerate the entire process efficiently. 136 As discussed earlier, ML with NLP methods allows for a rapid processing of extensive medical data and extracts the key information from the data that helps to enable new structural predictions and the repurposing of existing drugs. The study and results would be useful for finding effective medicines and vaccines for COVID-19. AI helps to speed up the entire pharmaceutical process, from disease prediction to drug development ( Figure 11). 55 ML and DL methods have the potential to assist drug development and biological research, but some limitations should be considered, such as knowledge about the selection of appropriate models, need for large training datasets, and high computational costs of training.

Machine Learning-based Search Engine
An ML-based search engine launched by AWS, named CORD-19, 64 can help researchers find coronavirusrelevant documents using natural language questions. The CORD-19 search offers an easy search interface in which researchers can make queries using natural language and produces detailed answers as well as source documentation. Firstly, the NLP tool is used to enrich the dataset, and the ML method is used to mine pertinent health information from unstructured text data about disease, treatment, and related information. The relevant information is then mapped to COVID-19 medical models using inference procedure and multilevel classification, involving different enrichment sources, and indexed in Amazon Kendra, 137 which is a precise search provision based on ML and DL approaches. Figure 12 illustrates the overall architecture of the CORD-19 search model. 138 It has robust natural language query capabilities that make it convenient to find and rank associated documents. The Amazon search model is developed and built from public data, available on the AWS COVID-19 dataset. 93 In the vast information world, NLP-based intelligent search engines provide an appropriate platform in which users must only input text without having to make judgments for the search results. Thus, it produces the correct documents and saves time and effort for users.

Discussion and Future Directions
A huge number of data has to be aggregated and analyzed to get useful information about the coronavirus. This aggregated data would be able to show the latest details about the onset Figure 10. A data-driven framework for discovering antibody sequences to treat coronavirus disease (COVID-19). and spread of the virus, as well as the latest findings related to the transmission of the disease. Thus, ML with big data analysis is used in health care to build a strong prediction tool for coronavirus outbreaks. On the other hand, NLP is used to extract information from unstructured data to enrich structured medical data. It is the matter of training a proper neural network on a prepared dataset that supports the machine in diagnosing the disease. ML models can also analyze chest x-ray and CT scans using deep neural networks that provide significant information on the patient's status, and the blood sample visual analysis can provide additional information on a patient's health and overall condition. Some companies have made efficient tools based on ML and DL to help health care professionals in detection and diagnosis of the virus, surveillance of contagious illnesses, and prediction of the virus outbreak efficiently, as well as researching and collecting a huge amount of data from various sources. Nowadays, the ML model with big data can be mined for the relevant information from social media for tracking the spread of coronavirus. Technology-based health care has great potential to improve patient outcomes, predict outbreaks of epidemics, forecast current virus infection, gain valuable insights, screen patients, map and prevent diseases, determine the chances of disease incidence, detect risk areas and stop the spread of the disease, deliver medical supplies, and foster the quality of daily life. According to the study, technology can help to accelerate research activities in health care as well as manage clinical and social services.
Two key points have to be noted in the literature, the datasets and the methodologies. The most crucial issue is that researchers have studied a limited volume of COVID-19 data in their studies. Increasing the data with many clusters will empower stable systems. The data were collected from many different sources, not a centralized repository. Both male and female cases were analyzed in COVID-19 datasets, in which a remarkable number of aged patients were available. No feature extraction and feature selection procedures were employed in many studies, and they utilized the  methods directly with raw data. Moreover, the ML and DL approaches have potential in combating the COVID-19 pandemic with limited resources. Many emerging technologybased solutions can be suggested to improve the public safety, security, and privacy concerns during this pandemic and in the future. The advanced technologies, including blockchain, 7 federated ML, 139 incentive mechanisms with big data, 140 combined AI and cloud computing, 141 5G wireless technologies, 142 and others, can offer effective solutions for confronting the COVID-19 pandemic. Researchers attempted to develop an effective approach by combining various models and methodologies with various technologies. Although these models show encouraging outcomes, they need further improvements with large datasets and clinical trials for employment in the health care sector.
The following research directions could be taken with AI technology to further the fight against future pandemics such as COVID-19: • Developing specialized, AI-based IoT architectures with big data to solve epidemic-related issues and build a comprehensive system to respond to COVID-19-like crises, such as prognostic modeling, real-time patient observing, and emergency operations in health care. • Using emerging ML to support decision making, drug discovery and treatment, medical devices, and the overall modernization of the medical industry. Hospital management and health care professionals are considering the advantages of applying AI in technology and keeping patients' information in private clouds, such as Google Cloud Platform. AI provides hospital management and doctors with ways to store patient data on clouds more securely and access health records via AI-infused technology. • Improving AI approaches for medical imaging analysis will help to examine patients' radiological and pathological reports accurately by machine, with doctors making proper decisions for treatment. Speech and text recognition using NLP could be employed for patient-doctor communication and capturing of clinical notes in remote treatment. • Collecting centralized, worldwide COVID-19 datasets could be beneficial for technological research to develop smart systems for prediction, diagnosis, forecasting, transmissibility, pathogenicity, and therapeutic methods during pandemics.

Conclusion
The outbreak of COVID-19 is observed keenly and this study concerns the reasons, impacts, and prevention policies at the very early stage of the pandemic. The outcomes of the study are summarized and a profound alignment with technology, especially AI and ML, is explored. This investigation intends to provide the current technological aspects of ML strategies that are appropriate to use in health care for facing the current pandemic situation. This article addresses the link between the technologies and the epidemics as well as the potential impacts of technology in medical services with the introduction of ML, DL, and NLP tools. AI has played an important role in confronting the coronavirus pandemic and supported researchers in developing systems to reduce human interaction in infected places, deliver services, and manage health emergency situations, but it needs ample resourcing and financial support to respond to public health threats. AI-related technology is being applied to track the span of infection rapidly, and the developed systems have potential to predict future outbreaks of the virus by early screening or diagnosis of patient data as well as inform the legal and ethical challenges of drug development in response to public health emergencies. Privacy and security are important concerns in health care data, and it is imperative to protect individuals' personal data through proper policies and authorization requirements. The designated methods should address the protection of data from pernicious attacks and ensuring that data is shared appropriately. The best, persuasive AI strategies, AI-based searching strategies, probabilistic models, and supervised learning are required to tackle future pandemic challenges and public health issues.


Gem5-X

The increasing adoption of smart systems in our daily life has led to the development of new applications with varying performance and energy constraints, and suitable computing architectures need to be developed for these new applications. In this article, we present gem5-X, a system-level simulation framework, based on gem-5, for architectural exploration of heterogeneous many-core systems. To demonstrate the capabilities of gem5-X, real-time video analytics is used as a case-study. It is composed of two kernels, namely, video encoding and image classification using convolutional neural networks (CNNs). First, we explore through gem5-X the benefits of latest 3D high bandwidth memory (HBM2) in different architectural configurations. Then, using a two-step exploration methodology, we develop a new optimized clustered-heterogeneous architecture with HBM2 in gem5-X for video analytics application. In this proposed clustered-heterogeneous architecture, ARMv8 in-order cluster with in-cache computing engine executes the video encoding kernel, giving 20% performance and 54% energy benefits compared to baseline ARM in-order and Out-of-Order systems, respectively. Furthermore, thanks to gem5-X, we conclude that ARM Out-of-Order clusters with HBM2 are the best choice to run visual recognition using CNNs, as they outperform DDR4-based system by up to 30% both in terms of performance and energy savings.

INTRODUCTION
The rapid growth of online services and increasing adoption of digital products have influenced all spheres of our lives. This increased digitization of the society has led to the emergence of new applications, which are deployed all the way from High Performance Computing (HPC) systems and cloud servers to mobile devices. Consequently, this situation has led to data centers consuming 0.8% (200 TWh) of the global energy in 2019 [24]. These new emerging applications, such as video analytics [2], autonomous driving, natural language processing, content recommendation [37], bioinformatics and genome sequencing [49], have different performance/energy requirements and are deployed on a variety of different platforms. To optimize these performance/energy constraints, a system-level simulator is required that is capable of simulating multi-threaded applications in a full-system environment with a complete operating system (OS) on a heterogeneous multi-core system. Furthermore, the simulator should also be capable of a detailed system-level profiling to identify the bottlenecks, therefore, helping in designing strategies and architectures to alleviate these bottlenecks.
In this article, we present gem5-X ("a gem5-based full-system simulator with architectural eXtensions") and demonstrate its capabilities to seamlessly analyze multi-threaded applications, enable various architectural extensions both for the compute and memory sub-systems, and explore various architecture parameters to have an overall performance-energy optimized architecture. The video analytics application is used as a case study to demonstrate the architectural and systemlevel characterization capabilities of the gem5-X framework. Moreover, gem5-X is generic enough to be used for detailed architectural analysis in any other application. Then, we present an optimization methodology based on local exploration and optimization for each individual kernel and then a global optimization for the whole application to have complete optimized system. Each kernel can have its own optimized architecture, which might be different from another kernel, but collectively all those locally optimized kernels make up the complete optimized application. Hence, we develop a heterogeneous system-level architecture and ensure that all the kernels and system components are working coherently with each other.
As previously mentioned, real-time video analytics, dubbed as the next-generation "killer-app" [2], is used in a wide range of domains, from video surveillance for security and monitoring, safety on construction sites, traffic monitoring and thermal monitoring to identify patients with fever, as well as for fire hazards [35,57]. Real-time video analytics is also used for autonomous drone navigation and rescue drone missions, e.g., during earthquakes, floods, or rescuing people at sea [25,36]. Autonomous drones are also being deployed for deliveries of parcels by many companies around the globe. The emerging market of smart autonomous cars, along with their Advance Driver-Assistance Systems (ADAS), use video analytics as one of the core components of the system to detect obstacles, traffic signs, and signals to navigate the car accordingly [45,46]. Hence, this "killer-app" is being deployed all the way from low-power edge devices to high-performance cloud servers.
Video analytics is a combination of two kernels, namely, video processing (or video encoding) along with image classification and detection. The video encoding kernel has to meet the performance and Quality-of-Service (QoS) constraints of processing at 24 frames-per-second (FPS) for a seamless user viewing experience. Image classification has real-time constraints in real scenarios such as in surveillance to identify and alert of a potential security breach or safety alert on a construction site, collision avoidance in drone navigation, and safety features and decision processing in autonomous cars and ADAS systems. These different use-cases of video analytics on different architectures, such as surveillance in the cloud and drone navigation on the edge, require to have a performance and energy-optimized architecture while meeting the real-time constraints. The optimized architecture will enable longer battery life for edge devices and lower energy costs as well availability of resources to serve more users on the cloud. Hence, a system-level simulator is required to optimize the performance and energy of the complete system for multi-threaded applications on different architectures, either on the edge or in the cloud.
The main contributions of the article are as follows: • We present the new gem5-X simulation platform, capable of simulating heterogeneous compute cores clusters and accelerators along with heterogeneous memory types such as the traditional DDR4 and 3D stacked memories. • The 3D stacked High Bandwidth Memory v2 (HBM2) memory model in gem5-X is updated with improved interleaving to uniformly distribute the memory traffic. It is then analyzed with the STREAM benchmark, giving insight into memory bandwidth (BW) scaling and the potential BW bottlenecks in the system. • A two-step architectural exploration and optimization methodology, capitalizing on our new gem5-X platform, is presented and showcased for a complex application like video analytics, composed of two kernels, namely, video encoding and image recognition using CNNs. • We explore and optimize the architecture with video analytics application as a case-study, using the two-step methodology. We demonstrate a fully functional heterogeneous architecture with ARMv8 in-order cores with an in-cache computing engine called BLADE [55] for video encoding, along with ARMv8 Out-of-Order (OoO) cores on a separate cluster in the system for visual recognition tasks. For video encoding kernel, we achieve 20% and 54% performance improvement when using ARM in-order cores with BLADE compared to baseline ARM in-order and OoO core system, respectively. Additionally, replacing the DDR4 by HBM2 further improves the performance and energy efficiency by 10%. In the heterogeneous system along with HBM2, we achieve both performance and energy benefits of up to 30% in comparison to an equivalent system with DDR4, which in fact cannot meet the performance requirements for visual recognition tasks on OoO cores.

RELATED WORK
State-of-the-art complex applications deployed across the range of compute devices from edge devices to servers in the cloud require new innovative and heterogeneous architectures to run optimally in terms of performance and energy efficiency [54]. Full system architectural simulators are required for fast architectural exploration with accurate estimations of performance and energy to give an insight during the initial design phase by enabling early deployment of the application on the simulation platform and reduce the time-to-market of new products [14,59].
Quite a lot of research has gone into developing architectural simulators, but each has its own shortcomings. Sniper [12] is a multi-core simulator with fast turnaround time, but only supports traditional x86 architectures. Simics [33] is another architectural simulator enabling applications to run on different hardware platforms. It is combined with Simflex [18] for timing information, but is limited to SPARC architectures only. Gem5 [9] is a full-system (FS) architectural simulator being widely used both in academia and industry, as it supports multiple ISAs, such as x86, ARMv7 ARMv8, MIPS, and ALPHA. In addition to a variety of ISAs, it also supports different CPU models for these ISAs, such as atomic, in-order, and OoO CPU models, as well as multiple caching protocols and coherences. On the memory side, it supports many traditional and emerging memories. Further, gem5 supports FS simulations via different Linux-based operating systems such as Ubuntu and Android, enabling applications to run as they would on a real platform. However, gem5 is quite slow with long simulation turnaround times. Also, not all configurations in gem5 are tested, and many models do not work in conjunction with others straight out-of-the-box (OOB).
Our initial version of gem5-X [50] was a simulation platform that enhanced gem5 with OOB architectural extensions and new memory models. It enabled seamless FS simulation with profiling support to analyze the speedup and gains of the functional blocks within the application as a result of new architectural extensions and optimizations. It also provided file-sharing support between the host machine and the simulated system using workload automation (WA), which is built into the Linux kernel provided for gem5-X. Therefore, it reduced the simulation turnaround time when debugging the application on a novel architecture within gem5-X.
In this new version of gem5-X, we support heterogeneous architectures, such as in-order and OoO cores along with custom accelerators like an in-cache computing engine [55]. In addition to heterogeneity at the compute side, it also supports heterogeneous memory types like DDR4 alongside 3D stacked HBM2, thus enabling a highly heterogeneous system with full Linux stack. To the best of our knowledge, this is the first work that simulates a complete Linux-based system with clustered heterogeneous compute cores (in-order and OoO) with in-cache computing engine along with 3D stacked HBM2 memory. In this article, we also perform the bandwidth (BW) analysis on HBM2 memory model in the new gem5-X using the STREAM benchmark [34] to validate that it provides the required BW in comparison to DDR4, as will be discussed in Section 4.3.
With increasing growth and popularity of cloud-based services, many of the emerging workloads can be deployed either on the cloud or on HPC systems. Gem5-X supports virtual machines (VMs) that can be deployed in the cloud infrastructure, as discussed in Reference [44]. It can also be used to explore and optimize architectures for HPC workloads. In particular, the work in Reference [49] demonstrates the use of gem5-X in enabling architectural exploration for the next-generation of genome sequencing HPC workloads.

VIDEO ANALYTICS APPLICATION
The gem5-X simulation framework supports full Linux stack and is generic enough to run and optimize any multi-threaded application, but to demonstrate its capabilities, a real-time video analytics application is used as a case-study in the context of this article. The choice of this particular application is due to two reasons. First, video analytics is deployed and used in different spheres of our daily life, e.g., in surveillance for safety and security, drone navigation, and autonomous cars. These different scenarios also present a challenge to system architectures because of different compute capabilities and energy constraints of the systems, all the way from the edge device to the cloud servers. The second reason for using video analytics as a case-study application is because of the complexity of the application itself and the fact that it is composed of two distinct kernels-video encoding and image classification using CNNs. This gives us the opportunity to Fig. 1. Video analytics application composed of two kernels running concurrently, i.e., video encoding using Kvazaar [60] and image classification/detection using CNNs.  [35,57] 640 × 480 5 Drone Navigation [25,36] 300 × 200 10 Autonomous Driving and ADAS [45,46] 640 × 480 15 exploit and demonstrate the capabilities of gem5-X in optimizing various memory and compute sub-systems to have an overall optimized architecture.

Video Analytics Application Structure
As depicted in Figure 1, real-time video analytics consists of two kernels running in parallel alongside each other, namely, video encoding/processing and image classification and detection [32,51]. Video encoding is used to encode the video stream and transmit it to the end-user or to store it in the cloud, e.g., in a security video surveillance system, the video is being streamed to a control room or stored in the cloud [35]; in case of a drone navigation system, it is being transmitted to the drone pilot [25]. In addition, to video encoding, the video frames from the video stream are being analyzed simultaneously by the image classification application. CNN-based image classification may be used for facial recognition, counting people, and potential hazard on a construction site in case of surveillance application [57]. It is also used for obstacle detection in drone navigation [25,36] and autonomous cars [45,46]. In a drone rescue mission, it can be used by drones to identify people at sea, in flood, or in the rubble during an earthquake. In essence, both the video encoding and image classification needs to run side-by-side for a fully functional video analytics. The video encoding needs to process 24 FPS for a seamless user experience. Simultaneously, the image classification needs to classify a number FPS depending on the application scenario. Table 1 summarizes the various application scenarios of video analytics application with different resolutions for video encoding and image classification/detection rate. In the following sections, we will present the video encoding and image classification kernels, both of which together make up the video analytics.

Video Encoding
Video encoding is an essential part of video analytics. It also has significant importance in online video streaming services, which contributed 58% of downstream internet traffic in 2018 [53], hence, the need for a performance-energy-optimized video encoding application. For this purpose, Kvazaar [60], a state-of-the-art open source High Efficiency Video Coding (HEVC) application,  compliant with H.265 coding standard, is used as real-time video encoding kernel. HEVC offers twice the compression of its predecessors, but at the cost of significantly increased computational cost [10].
In HEVC encoder, the most complex block is the motion estimation of the video, which plays a critical role in compression (and therefore, bandwidth) and quality. The goal of encoding is to serve videos to users in real-time, i.e., achieving a sustained frame rate of 24 FPS, regardless of the video resolution. Real-time HEVC encoding is achieved by means of thread-level parallelization of different blocks. In this work, we choose Kvazaar, as it comes with a wider range of parallel processing capabilities. Kvazaar is very well optimized for software, leaving limited headroom for further software-based optimization.
As discussed in Reference [50], FIR filter and the Sum of Absolute Transform Differences (SATD) are the two dominating blocks in Kvazaar that represent 21% and 26% of overall instructions executed, respectively. The remaining 53% of the computation is spread in chunks of less than 10% (in average 7%) instructions in other blocks. Hence, we will focus on defining a suitable architecture with gem5-X to optimize the execution of the FIR filter and SATD block in Kvazaar.

Image classification using CNNs
The video analytics includes video encoding, which is either sent to the end-user or stored in the cloud, and CNN-based visual recognition, as discussed in References [13,32,51]. CNN-based image recognition is now becoming a standard in both the industry and academia for computer vision tasks because of the impressive results they have achieved [30]. All the case study scenarios of surveillance, drone navigation, and ADAS use CNNs for the visual recognition part of the application, as discussed in References [25,57], and [46], respectively. Because of ample research in CNN architectures, there are a variety of CNNs to choose from [8]. The choice to deploy a particular CNN depends on various factors such as accuracy, inference time, memory requirements, computation complexity, and the size of the network. As the video analytics scenarios presented in Table 1 vary from being deployed on the edge device like in drone navigation all the way up to highend cloud servers like in a surveillance system. Therefore, the CNN we select should meet the memory, computational complexity, and performance requirements in all the scenarios.
MobileNet [20] is an efficient CNN architecture that can be deployed on the edge as well as in the cloud. It has a Top-1 accuracy of 70.4% on the Imagenet benchmark, which is better than other complex popular CNNs, such as GoogleNet, Alexnet, and Squeezenet. MobileNet-v1 is also quite small in size and complexity, and it is feasible to be deployed on the edge node. MobileNet has previously been used in surveillance [57], drone navigation [26,61], and autonomous cars and ADAS [29]. As most of the edge nodes are based on ARM architectures, we use the ARM Compute Library (ACL) [5] framework to deploy MobileNet. The limitation of MobileNet is that it can only process image sizes of up to 224x224 pixels, smaller than the case-study scenarios listed in Table 1. Hence, for these images to be processed by MobileNet, we perform image partitioning and resizing (if required) as shown in Figure 2. For the 300x200 pixel frame, we partition it into two 224x224 images. Since, the resolution of the resulting two 224x224 images is greater than the original image, we use the extra pixels to overlap the split images horizontally. This overlap allows us to detect objects that would have otherwise been split between images. The unused pixels at the bottom of the images are ignored and filled with black, as shown in Figure 2(a). For the frame resolution of 640x480, we first resize it to 640x448 pixels, so it can evenly be distributed vertically to 224 pixel size. After that, we partition it to six 224x224 pixel images, with the extra pixels being used to overlap the adjacent images to avoid splitting of objects between images, as shown in Figure 2(b). The partitioned images are then sent to MobileNet-v1 for image recognition.
MobileNet has been used in our case study, as it is a small, fast, and high accuracy CNN, which makes it feasible to be used both on the energy-constrained edge device, as well as on the servers in the cloud. It has also previously been used in our case-study scenarios in Table 1. However, any other CNN can be used instead of MobileNet, but the methodology we will use in the following sections will remain the same.

GEM5-X SIMULATION PLATFORM
The proposed gem5-X simulation platform [50] is the enabler for the architectural exploration and optimization of our case-study real-time video analytics application (cf. Section 3). The platform is composed of compute cores and memory devices. It simulates an ARMv8 64-bit ISA FS simulation with Linux kernel 4.3 and Ubuntu 16.04/18.04 as the OS. It also has application profiling support provided by the gperf profiler [17], with minimal overhead, to identify bottlenecks within the application executing in the simulator. In addition, gem5-X also supports 9P [28] over virtual IO [42], for fast modification and sharing of files between the simulated and host system.
The video analytics application has both high compute and memory requirements. Hence, we will now look into the compute and memory sub-systems of the gem5-X platform, as shown in Figure 3, with different exploration parameters highlighted in red.

Compute Sub-system
Gem5-X enables different strategies and aspects of the compute sub-system.
• Type of Cores: Gem5-X supports ARMv8 64-bit energy-efficient in-order cores as well as high-performance out-of-order (OoO) cores, both with a validation error of up to 4% as shown in our previous work [50]. We have added support for heterogeneous architecture simulation allowing both in-order and OoO cores to be simulated simultaneously, with different applications or application kernels being allocated efficiently to different core types to maximize performance, as well as energy efficiency, as described in Section 6.3.1. • Clustering: Gem5-X enables clustering of the compute cores, both for heterogeneous as well as homogeneous core types. Each cluster has it own last-level-cache (LLC). System architects can further expand the cluster implementation with each cluster having its own independent clock. Clustering enables independent applications or independent kernels of the application that do not share resources with other kernels of the application to execute on independent multi-core cluster with its own LLC.  • Accelerator Support: Gem5-X supports adding custom accelerators for specific tasks. To utilize the added accelerator, in FS mode simulation, one can either memory map or trigger it via a custom instruction. In case of memory mapping of the accelerator, it acts as an external device in the system. For an accelerator tightly integrated within the CPU pipeline, adding a custom instruction is the more feasible approach. For the video encoding part of our case-study application, an in-cache computational engine called BLADE [55] was added to the L1-D cache for increased performance and energy-efficient execution of Kvazaar, accelerating certain FIR filter and the SATD blocks within it, as presented in Reference [50]. The accelerator is supported in conjunction with heterogeneous compute cores as well as clustering. • ISA Extension: To support an accelerator or to add a new functional unit within the CPU core, gem5-X supports ISA extension, using the reserved op-codes of the original ISA specifications. To support BLADE in FS mode, we added a custom instruction using one of the unused opcodes in ARMv8 ISA specification [4]. The accelerator can be used by issuing in-line assembly calls to the new instruction in C/C++. • Core Count: Gem5-X enables many-core simulation with support for 256 cores simulation.
In this work, the number of cores we need are up to 31 cores, as discussed in Section 4.3.2.

Memory Sub-system
The memory sub-system is as essential as the compute sub-system, especially in the case of memory intensive applications, like the video analytics application. During video encoding, the raw video frames are read from memory and the encoded frames written back to it. During MobileNet CNN image classification, the input image, the weights, and the parameters of the network are both stored in memory and have to be accessed for each image inference. Hence, memories with high bandwidth help in alleviating the memory bottleneck in these memory-intensive applications. The 3D High Bandwidth Memory v2 (HBM2) [58] can achieve a bandwidth of up to 307.2 GB/s, enabled by multiple channels and the Through Silicon Vias (TSVs) that connect the 3D stacked DRAM banks. Gem5-X implements the HBM2 memory model with timing estimates as in Reference [50]. Thus, we have updated the memory interleaving among different channels for the HBM2 model to uniformly distribute the memory accesses among different channels. No additional support is required from the software perspective, thus enabling any application to be executed either on a traditional DDR4-or HBM2-based memory system in FS mode without modifying the software. Figure 4 shows the 3D stacked HBM2 with multiple channels in gem5-X. Each channel is 128 bits wide; hence, for 8-channels HBM2, the system bus is configured to be 1,024 bits wide.
Gem5-X also supports heterogeneous memory types such as traditional DDR4 along with 3D HBM2 simultaneously in the same system. Both memories are mmapped separately in the gem5-X configuration file. During the Linux boot-up, one of the memories is defined as the base memory, with the full address space available to the kernel. The other memory can then be allocated using mmap(), from within the application, if required. By default, allocations are done to the base memory defined during the kernel boot.

HBM2 Bandwidth Analysis
The gem5-X ARMv8 compute cores were already tuned to have a validation error of up to 4% in our previous work. In this article, we now perform the bandwidth (BW) analysis on HBM2 memory model in gem5-X to validate that it provides the required BW in comparison to DDR4 and also to check that the memory traffic is being uniformly distributed among different memory channels, as mentioned previously in Section 2. STREAM [34], which is a well-known memory bandwidth benchmark, is used for the HBM2 bandwidth analysis. Figure 5 shows the methodology for the analysis of HBM2 BW and its comparison to DDR4 in gem5-X using the STREAM benchmark by changing various architectural parameters in the system.

BW Analysis Methodology.
• Input: The STREAM benchmark is used as a benchmark on the ARM-based platform.
The input array size of the benchmark is set to 100M, which is equivalent to a memory requirement of 2.2 GB. The array size is chosen to have a tradeoff between memory utilization and simulation turnaround time, which increases linearly with the increase in array size. • Memory Channels: We run STREAM for HBM2 with 1, 2, 4, and 8 memory channels to see the effect of channel count on BW. • Memory Types: We use both HBM2 and DDR4 as memory types to compare their performance. DDR4 is used with 1, 2, and 4 memory channels, as the number of channels scale from 1 to 4 in commercially available DDR4-based systems when we move from low-power mobile devices to high-end server-class systems [21][22][23]. • Core Types: All the benchmark experiments are run for both ARMv8 in-order and OoO cores.
• Core Frequency: We repeat all the benchmark experiments at 1 GHz, 2 GHz, and 4 GHz core frequencies. The 4 GHz frequency is used just for scaling analysis, as we are not aware of any ARM cores operating at 4 GHz. • Core Count: As we want to analyze the effect of core count on BW, we benchmark core counts of 8 and 16 cores. • Cache Hierarchy: We also look into changes of the cache hierarchy, namely, the effects of no-LLC system in comparison to system with LLC. • Analysis: In addition to analysing the results after changing each architectural parameter discussed in this methodology, we globally analyze the results and run the experiment with a change in any architectural parameters necessary.

Bandwidth Analysis Results.
We now look into the BW analysis results of HBM2 in comparison to DDR4.
• Figure 6 shows the scaling of HBM2 bandwidth with the number of channels and core frequency, as compared to DDR4 with 1, 2, and 4 channels, for 8-core ARM in-order and OoO systems with a LLC (L2) of 1 MB. We see that the BW increases with the number of channels. HBM2 performs better than DDR4 by 5% for both in-order and OoO cores for same number of channels, whereas 8-channel HBM2 gives up to 34% and 48% more BW as compared to single channel DDR4, for in-order and OoO cores, respectively. Moreover, 8-channel HBM2 gives up to 12% and 19% more BW as compared to 4-channel DDR4. BW scales with the core frequency for 8 in-order cores as in Figure 6(a). We also see that OoO cores utilize much more of the available BW as compared to in-order cores at the same frequency. However, the BW scaling with frequency is low after 2 GHz. To investigate this further, we run STREAM with higher number of cores as discussed below. • We run the STREAM benchmark for 16 ARM in-order and OoO cores, both with LLC. Figure 7 shows that BW saturates for OoO cores at higher frequency, and BW utilization of in-order cores converge to that of OoO cores. If we compare the BW of 8-OoO cores in Figure 6(b) to that of 16-OoO cores in Figure 7, the BW does not scale with the increase in the core count of OoO cores. However, it does scale with the number of in-order cores. The BW of 8-channel HBM2 is 44%-46% higher than 1-channel DDR4 and 17%-19% higher than 4-channel DDR4. Hence, we limit ourselves to 8-channel HBM2 BW scaling analysis for now. • Through our analysis of the data path between the compute cores and memory, we observe that the LLC is the bottleneck on the available BW, which explains why OoO cores do not exhibit a linear scaling with the number of cores. However, the reduced bandwidth scaling is also an indication of better performance, as the memory is being accessed less frequently due to the caching effects of LLC. However, we remove the LLC from the system and run the STREAM benchmark again. As shown in Figure 8, the BW scales with number of cores  both for in-order and OoO cores without the LLC. From Figure 8(a) and Figure 8(b), it is also evident that OoO cores can utilize almost 2× more BW as compared to in-order cores. Also, these figures show that the BW utilization depends on the core count and type of cores. Thus, the BW provided by HBM2 is available, the greater the number of cores, the more its utilization. However, we observe that the BW utilization does not scale linearly in relation to the core frequency and remains almost constant. • We investigate the L1 cache to see if it is causing any bottlenecks in relation to BW scaling with frequency. We change the size of miss-status-holding-registers (MSHR) 1 from 4 to 10. However, there was no change in BW for in-order cores, and around 16% BW improvement for OoO cores when using 10 MSHRs instead of 4. However, the change with frequency scaling was still quite constant. Hence, the reason for almost constant BW being accessed across different frequencies is not due to L1, but results from the fact that the core frequency to issue memory request is high compared to HBM2 serving those requests. Thus, whenever the request goes to HBM2, the cores are stalled waiting for memory to respond to those read/write accesses. As STREAM is a memory intensive benchmark, changing the core frequency does not improve the stressed BW of the memory.

Power Models and Area
In addition to performance analysis using gem5-X, we are also interested in the power and energy consumption of the systems. However, we did not use gem5 or McPAT power model, as proposed by References [52] and [11], respectively, as they both are for ARMv7 32-bit ISA, whereas we are using ARMv8 64-bit cores. For the CPU energy analysis, we use the power model for 28 nm CMOS bulk technology node for ARM 64-bit in-order and OoO cores proposed in References [44] and [50]. This power model includes the core active, static, and wait-for-memory (WFM) energy, LLC read/write, and static cache energy. For the memory power models, power values as reported in References [31] and [43] are used for DDR4 and HBM2, respectively. Counters in gem5-X statistics  such as active CPU cycles, WFM cycles, cache read and writes hits, and main memory accesses are used for power modeling.
For area estimates, we use the values reported in References [16] and [15] for ARM OoO and in-order cores, respectively.

ARCHITECTURAL EXPLORATION AND OPTIMIZATION METHODOLOGY
We propose a methodology to explore and optimize architectures for applications that are composed of multiple kernels such as the video analytics application or multiple independent applications allocated simultaneously on a platform. Figure 9 shows the two-step methodology for architectural exploration and optimization of this multiple kernel scenario. The first step is the local architectural optimization for each kernel based on its own bottlenecks and compute/memory requirements. The optimization is on both compute and memory fronts, exploiting their respective architectural parameters as presented in Figure 9, Step 1, and also discussed in Figure 3 in Section 4. Once all the kernels are independently architecturally optimized, we co-allocate them together and globally optimize the architecture for all kernels, if further necessary, as in Figure 9, Step 2. The optimization strategy in Step 2 is the same as for local optimization in Step 1, both on compute and memory side, hence leaving limited room for further optimization, unless there are conflicting architectural requirements between different applications. In such a scenario, the focus is on minimum performance requirements or QoS for each kernel and then further explore the architecture.
Our case-study video analytics application is composed of two kernels, video encoding and CNN-based image recognition. Hence, the complexity of the methodology in Figure 9 is linear. However, if there is an application with more than two independent kernels, i.e., N > 2, then the complexity for the exploration methodology will still be linear. In such scenario, we would coallocate all independent architecturally optimized kernels as in Step 2, because they do not share any resources except for at shared cache or the main memory level. The shared caches can be made independent and private for each kernel with the clustering approach.

COMPUTE RESOURCE ANALYSIS AND OPTIMIZATION FOR REAL-TIME VIDEO ANALYTICS
To have an optimal architecture in terms of performance and energy efficiency for our case-study video analytics, we use the two-step exploration methodology discussed in Section 5. Therefore, as in Step 1 of the methodology in Figure 9, we do a separate architectural exploration and optimization for Kvazaar and MobileNet. Then, we co-allocate them together on a single platform as a video-analytics application and perform global optimization as in Step 2 of the methodology.
Gem5-X: A Many-core Heterogeneous Simulation Platform 44:13  Table 1, we need to meet 24 FPS requirement. As discussed in Section 3.2 and in Reference [50], FIR filter and SATD are the two dominating blocks in the Kvazaar encoder. Hence, we will focus on optimizing the architecture for these significantly contributing blocks to have an architecture optimized for the overall application.

Experimental Setup.
We use ARMv8 64-bit cores booted in Ubuntu 16.04 OS. As a starting point, gem5-X is configured as the ARM JUNO platform [3], which is a validated configuration as in our previous work [50]. The initial configuration is summarized in Table 2.
Our experimental setup is based on general-purpose compute cores (ARMv8 cores) and not ASICs, as quite a lot of effort and time is required for the development of an ASIC. A softwarebased solution can ease this with just downloading and optimizing the code and running it on available general-purpose compute system. Furthermore, applications developed to run on ASIC cannot be updated/upgraded if there are new improvements to the algorithm. However, CPU-based system allows for updating the currently deployed application with updated or new algorithms on the same platform.

Profiling, Bottlenecks, and Sweeping Parameters.
We profiled Kvazaar using Valgrind [38] on ARM JUNO and gperf [17] in gem5-X. On the computational dominancy end, FIR and SATD blocks were dominating with 21% and 26% instructions, respectively, and hence they are potential candidates for optimization. The remaining 53% was distributed among other blocks with contributions of less than 10% and hence not considered for optimization. This profiling data was consistent for both ARM in-order and OoO cores and also both in gem5-X as well as ARM JUNO.
On the memory side, the L1 cache miss rate was 4.8% and 5.2% for FIR and SATD, respectively, indicating high locality. Hence, we sweep the sizes of both L1 and LLC from 8 KB to 128 KB and 512 KB to 16 MB, respectively, to find the optimal size. The L1 size of 32 KB along with a LLC of 1 MB is found to be optimal in terms of both performance and area, as also discussed in Reference [50].

BLADE -In-cache Computing
Engine. BLADE is a hardware extension for CPU caches that enables computing directly within the cache. Blade operations are performed by first precharging the bitlines of the cache subarray, as illustrated in Figure 10(a). Then, two wordlines are activated simultaneously, thus allowing the contents of two rows of bitcells to be connected to the bitlines, as illustrated in Figure 10(b). The bitlines are discharged according to the contents of the bitcells, resulting in an and operation on the bitline and a nor operation on the inverse bitline. These signals can be further combined via a nor gate to achieve a xor operation. Finally, further processing allows complex operations such as addition and multiplication to be performed [55,56]. The operation results are then written back to the cache. Application runtime is improved in two respects: first, data movement is reduced, and second, Single Operation Multiple Data (SIMD) operations can be performed on multiple operands simultaneously, for example 128 1-byte operations in a cache with two subarrays and 64-byte wordlines. It also reduces energy consumption, as the computation is performed in the cache, hence saving energy both on the internal bus transactions as well as static energy for the idle core when the data is being fetched.
BLADE provides acceleration primarily for applications that exhibit high cache locality, as it performs SIMD operations on many physically successive operands. Since FIR and SATD perform simple operations with high cache locality, they are potential candidates for BLADE acceleration, as discussed in Reference [50], and have thus been modified at the source-code level to support BLADE operations. BLADE has been incorporated in the L1 cache for ARM in-order cores.

Architecture Exploration -Results and Analysis.
There are two video resolutions, 300x200 pixels and 640x480 pixels, that cover all three-video analytic case-study scenarios, namely, surveillance, drone navigation, and autonomous cars, as presented in Table 1. Therefore, for the architectural exploration and optimization of Kvazaar, we will be using these two resolutions. Our baseline architecture will be as in Table 2, with ARMv8 64-bit in-order cores with the core count dependent on video resolution to meet the 24 FPS requirement. BLADE will be used as an accelerator along with ARMv8 in-order cores. We will also be comparing the performance to ARMv8 OoO cores.
• 300x200 Resolution: Figure 11(a) shows the performance and energy benefits of ARM 4-core in-order system with BLADE and ARM 4 OoO cores in comparison to the baseline of ARM 4 in-order cores, all operating at 2 GHz and DDR4 as main memory, while processing 300x200 resolution video at 24 FPS or more. We observe that 4 in-order cores with BLADE are around 8% performance efficient, with a negligible energy overhead of 1% as compared to in-order system without BLADE. OoO cores, however, improve performance by 45% but at the cost of consuming 44.44% more energy than 4 in-order cores. Hence, we scale the OoO core count to 2-cores, to just meet the 24 FPS and not more. Also, we select 4 in-order cores with BLADE as our reference and compare their energy and area to 2-OoO cores. Now both systems have the same performance to encode at 24 FPS, but 2 OoO cores consume 45% more energy along with an area overhead of 31%, as depicted in Figure 11(b). Therefore, a 4-core in-order system with BLADE is the optimal architecture for 300x200 resolution video.
Gem5-X: A Many-core Heterogeneous Simulation Platform 44:15 Fig. 11. Performance, energy, and area comparison for Kvazaar video encoding of 300x200 resolution video. Fig. 12. Performance, energy, and area comparison for video encoding of 640x480 resolution video.
• 640x480 Resolution: For 640x480 resolution video, we start with a core count of 4 cores for ARM in-order, in-order with BLADE, and OoO cores at 2 GHz. As shown in Figure 12(a), 4 in-order cores with BLADE perform 20% better than in-order cores without BLADE in terms of FPS, along with 18% less energy consumption. The performance and energy benefit for in-order cores with BLADE is more in comparison to 300x200 resolution video, as a larger dataset exploits more cache capacity and provides more opportunities for using BLADE vector processing in the cache. The OoO cores improve performance by 45%, but consume 44% percent more energy in comparison to in-order without BLADE, similar to 300x200 resolution video. However, with 4 cores none of these architectures meet the 24 FPS requirement. Hence, we increase the core count until the 24 FPS requirement is met. For in-order cores without BLADE, we increase the number of cores to 12 to reach 24 FPS, whereas, with BLADE only 10 cores are necessary, indicating a performance benefit of 20%. The number of OoO cores at 2 GHz required to encode 24 FPS is 6. While processing 24 FPS for a 640x480 resolution video, we compare the energy and area required for OoO cores in comparison to in-order cores with BLADE. We find that 10 in-order cores with BLADE are 54% more energy-efficient compared to 6 OoO cores, and at the same time taking 43% less area as well, as shown in Figure 12(b).
The BLADE in-cache computing engine reduces the energy consumption, as it operates on larger dataset using the single-instruction-multiple-data (SIMD) approach as discussed in References [50,55]. Furthermore, it reduces the data movement between cache and the functional units in the CPU, as the operations are performed in cache, which reduces the energy consumption. For the video encoding kernel, Table 3 shows the core and memory power for 300x200 and 640x480 resolution videos while meeting the 24 FPS requirement. Here, we see that the energy reductions mainly come from the core power. In-order cores with BLADE in-cache computing has the least power, as compared to OoO cores, as well as the in-order cores without BLADE. In summary, this situation leads to low-power ARM in-order cores with BLADE to match the performance of high-performance ARM OoO cores with 54% less energy budget. Therefore, for the video encoding portion of video analytics, we will be using ARM64 bit in-order cores along with BLADE in-cache computing engine, as it produces the best energy efficiency and area occupancy, while achieving the required 24 FPS. All the cores in the system are 100% utilized. Hence, there cannot be any resource sharing.

MobileNet
MobileNet [20] is a state-of-the-art CNN used for image classification, with low computational cost designed for deployment on mobile and edge devices. As image classification and detection are necessary for video analytics, we will be using MobileNet for this purpose. The ARM ACL [5] framework is used to deploy an ARM optimized version of MobileNet, which we use in our gem5-X-based experiments to look into further architectural exploration and optimization choices.
MobileNet allows input image size of 224x224 pixels, whereas our three case-study scenarios in Table 1 have 300x200 and 640x480 pixel resolution. As discussed in Section 3.3, each 300x200 and 640x480 frame is split into 2 and 6 images of 224x224 pixels as inputs to MobileNet, respectively. Therefore, the resulting FPS requirement for MobileNet is higher, as shown in Table 4.

Performance Comparison -CPU vs. GPU.
GPUs are increasingly being used both in training and inference of the CNNs architectures due to their high-performance capability of processing vectored data, making them a perfect fit for CNNs [27,62]. Then, as our case-study scenarios involve edge devices, we looked into energy-efficient GPUs for mobile and edge computing.
Nvidia Jetson Nano is low-power state-of-the-art GPU designed for artificial intelligence (AI) tasks on embedded and edge devices [41]. We compare the performance and energy consumption for MobileNet inference between ARM in-order cores, OoO cores, and Jetson Nano. The Nvidia Jetson Nano platform [40,41] comes with 128 Nvidia Maxwell GPU cores integrated with 4 ARM Cortex A-57 cores. Furthermore, the Nvidia Jetson Nano has better software support, as compared to other platforms such as ARM Mali GPU [7] and Ethos NPU [6]. It can run the networks implemented in Tensorflow directly and has the optimized versions of the networks in TensorRT [39], an optimized framework to deploy CNNs on Nvidia GPUs. MobileNet is deployed on Jetson Nano using the Nvidia TensorRT framework [39]. The energy statistics on Jetson Nano are collected using the Nvidia tegrastats utility. Jetson Nano is set to its high-performance mode (10 W mode) with its 128 GPU cores operating at 921 MHz. The CPU and GPU in the Jetson Nano are in separate power domains [40], and in high-performance mode, the power rails VDD_CPU and VDD_GPU for CPU and GPU, respectively, are both set to 1.322 V. These power rails only account for the CPU and GPU compute cores of Jetson Nano and not any other peripherals or I/Os, as they are in separate power domains. Both ARM in-order and OoO cores are configured in gem5-X to operate at 2 GHz with 32 KB L1 instruction and data cache. The LLC is configured to 1 MB for 4 cores. As we will discuss later in Section 6.2.2, MobileNet performance scales linearly with multiple threads on multi-core system up to 4 cores. Hence, for an 8-core simulation, we launch two instances of Mobilenet, each using 4 cores independently, so the performance scales with core count. Thus, we use a clustering approach that we discuss in detail in Section 6.2.2. Then, HBM2 is used as memory instead of DDR4, which is an architectural optimization that we discuss in detail in Section 6.2.3. Figure 13 shows the performance in terms of FPS achieved by MobileNet on different architectures. This figure shows that ARM 8 in-order and 4 OoO cores surpass the performance of Jetson nano by 3.3% and 11%, respectively. Hence, 4 OoO cores achieves the highest performance. The reason as to why the lower number of ARM CPU-based system matches or outperforms Jetson Nano comes from the fact that ARM CPU-based system is operating at 2 GHz, whereas, Jetson Nano GPU-based platform with 128 GPU cores is operating at 921 MHz. Regarding energy comparison, we computed the energy for processing 21 MobileNet images. Jetson Nano consumes the highest energy, as can be seen in Figure 13, whereas OoO cores are the most energy-efficient. OoO cores are 27% more energy-efficient as compared to Jetson Nano and 6% in comparison to 8 in-order cores. The energy of 8 in-order cores is the same as 4 in-order cores, as 8 in-order cores achieve double the FPS (half the execution time) in comparison to 4 in-order cores. Overall, OoO cores with HBM2 are the best both in terms of performance and energy efficiency for processing MobileNet. Jetson Nano consumes more energy in comparison to ARM CPU-based system, because the high number of GPU compute cores (128 cores) in Jetson contribute largely towards the high energy consumption [19]. Secondly, the energy for the Jetson Nano platform includes the GPU energy as well as the 4 ARM Cortex-A57 cores, which are mostly idle and not used for the actual computation of the MobileNet CNN, but are used as the host cores for the GPU. As these cores are mostly idle, their static energy adds to the total energy of the Jetson Nano platform.

MobileNet Scaling and Clustering.
To achieve higher FPS, e.g., 90 FPS for an ADAS system, more cores are required to process the images, as well as the use of multiple threads with one thread per core. Therefore, we investigate how MobileNet scales with the number of cores.
We find that FPS scales linearly up to 4 cores, however, after that, the performance does not scale up with core counts and number of threads. This is due to the fact that, since MobileNet is a    Table 1. lightweight network, the cost of sharing data among higher number of cores affects the performance per core, thus suppressing the overall performance.
To overcome this issue of scaling, we launch multiple independent MobileNet instances, with each instance using a maximum of 4 cores. Hence, we instantiate 4 core clusters in gem5-X, each with its own LLC, so there is no thrashing in the cache. The clustered architecture with three clusters is shown in Figure 14.

Architecture Exploration -Results and Analysis.
For the compute side optimization of Mo-bileNet, we already concluded in Sections 6.2.1 and 6.2.2 that ARM OoO cores are the best in terms of compute core performance, and that to use clustering for parallel processing and launching separate instance of MobileNet on each cluster.
Next, regarding the memory side exploration, we consider 8-channel high bandwidth memory (HBM2), as CNNs are memory-intensive applications due to weight and bias accesses of the network. For our three-video analytics application case-study scenarios, we need to meet the FPS requirement for the visual recognition portion, as in Table 4. Hence, we run experiments with ARM OoO cores at 2 GHz, with the architecture shown in Figure 14, both with DDR4 and HBM2 memory types. Figure 15 shows that 4 OoO cores with HBM2 meets the FPS requirement for drone navigation and surveillance scenarios, whereas, 4 OoO cores with DDR4 (1 to 4 channels) only meet the FPS requirement for drone navigation. However, using HBM2 instead of DDR4 results in energy benefit of up to 21%, as shown in Figure 16(b). For ADAS and autonomous smart cars, where the requirement is 90 FPS, we launch three instances of MobileNet, each on a 4-core cluster. Figure 15 shows that 12 OoO cores with HBM2 meet the 90 FPS requirement. Again, 12 OoO cores with DDR4 (1 to 4 channels) fail to reach the required FPS. We also launch the experiments with 8 OoO cores with two MobileNet instances and see that it scales well with our clustering approach, from 4-core to 8-core to 12-core systems.
Finally, we compare the percentage performance and energy benefit of using 8-channel HBM2 instead of DDR4, with number of channels varying from 1 to 4, for MobileNet. Figure 16(a) shows that the performance benefit varies from 8% to 28%, as it scales with the number of cores as well as with the number of DDR4 channels. Similarly, as shown in Figure 16(b), there are energy savings of up to 32% for 12-core system when using HBM2 instead of single channel DDR4. The energy savings when using HBM2 instead of 4-channel DDR4 is between 16% to 20%, depending on the number of compute cores.
Therefore, ARM OoO core system with 8-channel HBM2 is the best-performing system both in terms of performance and energy efficiency for MobileNet-based visual recognition tasks.

Video Analytics
Once the Kvazaar video encoding and MobileNet visual recognition kernels are independently optimized for performance and energy efficiency, we move to Step 2 of the methodology in Figure 9. Then, the global architectural optimization and exploration is performed with all the application kernels co-allocated on the same platform, running in parallel. This can result in heterogeneous architectures with different clusters, as each locally optimized architecture might be different from one kernel to another, which share the crossbar interconnect and memory. Figure 17 shows the heterogeneous architecture for the complete video analytics application. There is a separate compute cluster for Kvazaar, with ARM inorder cores and ISA extensions to use the in-cache computing engine, BLADE, integrated into the L1-D. Similarly, there are multiple clusters for MobileNet with 4 ARM OoO cores per cluster. The number of clusters depends on the required FPS. All of the clusters have their own LLC. HBM2 is used as the main memory in the system, as it offers more BW compared to DDR4 and it is necessary for MobileNet to achieve the required FPS. All the cores and clusters are 100% utilized for both the Kvazaar and MobileNet kernels. Hence, there will be no resource sharing between different kernels in terms of compute cores and their components, as it will not benefit the application performance. We will look into the performance and energy benefits of using HBM2 for Kvazaar as well as the overall system in the next section.    Tables 1 and 4 and improve the energy efficiency as much as possible. As there are three scenarios, one with a resolution of 300x200 pixels (drone navigation) and two with 640x480 pixels (surveillance and autonomous cars), we will explore and discuss the results according to the video resolutions.

Heterogeneous Architecture.
• 300x200 resolution: We concluded in Section 6.1.4 that 4 in-order cores at 2 GHz with BLADE in-cache computing is the optimal architecture to meet 24 FPS requirement for the Kvazaar video encoding part of video analytics. For visual recognition using MobileNet, 4 OoO cores at 2 GHz were optimal in terms of performance and energy to meet the FPS requirement of 20 FPS for drone navigation, with HBM2 as the main memory. Hence, the architecture in Figure 17 will have one Kvazaar cluster with 4 cores and LLC of 1 MB and one Mobilenet cluster of 4 OoO cores. Figure 19(a) shows that performance benefit of 10% and 16.6% for Kvazaar and MobileNet clusters, respectively, when HBM2 is utilized instead of single-channel DDR4. The FPS values for both clusters with HBM2 and DDR4 are shown in Figure 18. The performance benefit is approximately 9% for both clusters when using HBM2 instead of 4-channel DDR4. We also observe an energy benefit of 6% and 20.7% for Kvazaar and MobileNet, respectively, when HBM2 is compared to single-channel DDR4. When compared to 4-channel DDR4, the energy benefit is 2.5% and 18% for Kvazaar and MobileNet, respectively. The FPS requirement of Kvazaar is that of 24 FPS, whereas that of MobileNet is 20 FPS, and we are above these minimum requirements for both applications, as shown in Figure 18. Moreover, we also compare our proposed architecture with HBM2 to commercially available ARM big.LITTLE Gem5-X: A Many-core Heterogeneous Simulation Platform 44:21 Fig. 19. Percentage performance and energy benefit when using HBM2  in comparison to DDR4 with different number of channels and Hikey960 platform for video analytics of 300x200 resolution video for drone navigation.
Hikey960 platform [1], which has 4 in-order and 4 OoO ARM cores. As shown in Figure  18, the Hikey960 achieves the least FPS, leading to 18% and 35% performance benefit of our proposed architecture using HBM2 over Hikey960, for Kvazaar cluster and MobileNet cluster, respectively, as shown in Figure 19(a). With regards to energy, we achieve 18% and 43% energy savings over Hikey960, for Kvazaar and MobileNet clusters, respectively, as shown in Figure 19(b).
To further optimize the architecture, we use Step 2 of our optimization methodology to optimize for energy efficiency, as we have already met the performance requirements. Both kernels scale linearly with frequency. Hence, we reduce the frequencies of both clusters to meet minimum performance requirements, therefore, enabling more energy savings. Reducing the frequency of Kvazaar 4-core cluster from 2 GHz to 1.72 GHz results in an energy saving of 32% while achieving 24 FPS. Similarly, for the MobileNet cluster, we reduce the frequency from 2 GHZ to 1.2 GHz, resulting in an energy saving of 59%. • 640x480 resolution: For the 640x480 resolution, two case-study scenarios exist. First, one of video surveillance with MobileNet requirement of 30 FPS. Second, autonomous cars and ADAS with MobileNet are used for image recognition with a requirement of 90 FPS. The video encoding for both scenarios is required to execute at 24 FPS. Hence, as discussed in Section 6.1.4, a 10-core in-order cluster at 2 GHz with BLADE is instantiated for Kvazaar, with HBM2 as main memory instead of DDR4. For MobileNet clusters, a 4-core OoO cluster at 2 GHz meets the 30 FPS requirement and three 4-core clusters meet the 90 FPS requirement, as discussed in Section 6.2.3. The performance benefit of 10-core Kvazaar cluster varies from 6.5% to 10.5% when using HBM2 instead of single-channel DDR4, while meeting the 24 FPS requirement, as shown in Figure 20(a), and between 2.8% to 4.3% when comparing HBM2 to 4-channel DDR4. We can see that the percentage benefit of the Kvazaar cluster for 640x480 resolution video, when HBM2 is being used instead of DDR4, is less as compared to 300x200 video resolution as in Figure 20(a). The reason being that, the higher the resolution, the more efficiently BLADE in-cache computing and caching is used as compared to lower resolution video. Hence, lower BW utilization of main memory and less performance benefit. The corresponding energy savings of using HBM2 for the Kvazaar cluster is between 6% to 10% and 2.6% to 3.3%, in comparison to single-channel and quad-channel DDR4, respectively, as depicted in Figure 20(b). For the MobileNet clusters, the performance as well as energy efficiency scale, as we increase the 4-core clusters from 1 to 3 for surveillance and ADAS, respectively. The  performance improvement of using HBM2 is up to 30% with a corresponding energy benefit of 30% in comparison to single-channel DDR4. When compared to 4-channel DDR4, the performance and energy benefits of using HBM2 are up to 12% and 10%, respectively, for the MobileNet clusters. Comparing the results of MobileNet cluster in Figure 20 to that of in Figure 19, it can be observed that the performance and energy savings are higher for three 4-core MobileNet clusters when it is co-allocated with Kvazaar than when it is standalone. This is because, when more applications are being allocated on a single platform sharing the same memory, the memory BW requirements increase, which HBM2 serves better than DDR4.
The overall energy reduction when using HBM2 instead of DDR4 is due to the fact that HBM2 not only has lower energy per access in comparison to DDR4 [31,43], but also enables faster memory access via eight memory channels. The faster access in turn implies less stalling of the processor pipelines when waiting for data, resulting in overall energy reduction. Table 5 summarizes the core power, memory power, and execution time per frame for both kernels of video analytics in drone navigation, surveillance, and ADAS scenarios. We see that the runtime to process each frame is less when using HBM2 instead of DDR4. Second, the memory power when using HBM2 is less as compared to when using DDR4. Therefore, lower memory power and fast processing time for each frame (both for video encoding and MobileNet kernels) contributes to the energy benefits we see when using HBM2 instead of DDR4.
In these case-study scenarios for surveillance and autonomous cars, we do not use Step 2 of the methodology, as the performance requirements are just met with the proposed architectures. Thus, we cannot further reduce the frequency to optimize for energy, as we did in the case of drone navigation, because this will degrade the performance below the minimum requirement.
In summary, capitalizing on our new gem5-X simulation platform, we propose an optimized heterogeneous architecture for a video analytics application, composed of ARM in-order cores along with BLADE in-cache computing engine being used for the video encoding portion and ARM OoO core clusters for CNN-based image recognition with HBM2 as the main memory. This heterogeneous architecture meets all the performance requirements and at the same time is energy-efficient in all the three case-study scenarios. In this case study, Step 2 of the methodology was used to optimize for energy by reducing the frequency for drone navigation scenario. Other than this, as the two kernels of the application do not give rise to any conflicting requirements, hence, no tradeoffs have to be made. Moreover, the methodology is generic and if an application requires further architectural exploration during Step 2, it can be performed. For instance, in case of different kernels of the application running on different cores, when allocated together on a single system, they might all share the same LLC. In that case, the strategy to follow in Step 2 would be to optimize LLC size to fit the working dataset of all kernels. In another instance, the memory requirement for individual kernels might be sufficient, but when co-allocated, the application might run out of memory. Thus, in Step 2, we will select appropriate memory size, forcing us to even change the memory technology being used, as some memories like the 3D stacked HBM2 are limited in terms of the memory size, as compared to traditional DDR4 memories. Therefore, Step 2 of the methodology can be used to optimize the architecture for the compute and memory sub-systems in cases when conflicting requirements arise after co-allocating all the kernels of the application together, in addition to the tuning of core frequencies for optimal performance and energy. Furthermore, the two-step methodology is generic and not specific to any application or application domain. It can be used to explore and optimize architecture for any application domain. The advantage of using the methodology for a general-purpose CPU-based system is that it gives an optimized architecture for the application domain and is not specific to an application. This enables updating the application or replacing the application or one of its kernels with a better application or kernel, which is not possible in ASICs, as they are application-specific. This has been demonstrated by using MobileNet for image classification kernel in this work, instead of AlexNet, which was used in our previous work in Reference [50]. From Figure 19, we can see that using an 8-core system with 4 cores allocated to Kvazaar and 4 cores for MobilNet, the performance benefit of using HBM2 instead of DDR4 is up to 10% and 16.6% for Kvazaar and MobileNet, respectively, in comparison to 7.72% and 8.35% performance benefit for Kvazaar and AlexNet, respectively, in the previous work in Reference [50], in a similar 8-core system.

CONCLUSION
In this article, we have presented the new gem5-X simulation platform to enable exploration of many-core heterogeneous architectures to optimize performance and energy consumption in new emerging dynamic applications, such as CNNs for image classification and multi-view video encoding. Compared to gem5, the latest version presented in this article of gem5-X supports validated ARMv8 in-order and OoO cores in FS mode with a validation error of up to 4%. Also, gem5-X supports multiple heterogeneous ARMv8 cores along with heterogeneous memories including DDR4 and the new 3D stacked HBM2 in FS mode, completely integrated and tested, which gem5 does not support straight out of the box. In addition to heterogeneous compute cores, gem5-X enables exploration of clustering configurations of compute cores. Moreover, gem5-X supports accelerators to be integrated within the system like the BLADE in-cache computing engine by capitalizing on an ISA extension support. Furthermore, in comparison to gem5, gem5-X enables profiling support using the gperf profiler, supports advanced check-pointing to help reduce the simulation turnaround time, and comes with workload-automation (WA) to enable file sharing between host machine and the simulated system. All in all, in this work, we have proposed a new two-step architectural exploration and optimization methodology of new many-core architectures for new dynamic applications and benchmarks.
Using the system configuration capabilities of our new gem5-X framework and exploration methodology for many-core systems, we have analyzed the benefits of HBM2 vs. DDR4 in the STREAM benchmark. Then, real-time video analytics was used as a complete case-study application to explore and optimize architectures in different execution scenarios. In particular, we have used three scenarios of video analytics, namely, surveillance, drone navigation, and autonomous cars and ADAS, for architecture exploration and optimization, as each has different performance requirement. In the end, thanks to gem5-X and our exploration methodology, an optimal clustered heterogeneous architecture with ARM in-order cores cluster along with a BLADE in-cache computing engine embedded within L1-D cache was defined as optimal option to process the video encoding kernel, as well as an OoO core cluster was included to process the CNN-based image classification kernel. Overall, gem5-X allows to select the right number of clusters and their operating frequencies for different case-study scenarios according to the performance requirement in each case. Furthermore, on the memory side, gem5-X proved that the use of HBM2 for the video analytics application led to both performance and energy benefits of up to 30%, compared to similar DDR4-based system. Hence, we demonstrated that using gem5-X enables a truly complete and fast design space exploration for many-core architectures.
The gem5-X simulation framework is open-sourced to the community, enabling out-of-the-box, fast simulation of many-core ARM 64-bit heterogeneous architectures with innovative architectural extensions. It is readily available for download online in this link: https://esl.epfl.ch/gem5-x. [48]. A technical reference manual for gem5-X has also been published online [47], which includes a quick-start guide as well as instruction on how to use different architectural extensions and support enhancements in gem5-X.