extfieldsofstudy:
Computer Science
Medicine

text:
COBERT: COVID-19 Question Answering System Using BERT

In the current situation of worldwide pandemic COVID-19, which has infected 62.5 Million people and caused nearly 1.46 Million deaths worldwide as of Nov 2020. The profoundly powerful and quickly advancing circumstance with COVID-19 has made it hard to get precise, on-request latest data with respect to the virus. Especially, the frontline workers of the battle medical services experts, policymakers, clinical scientists, and so on will require expert specific methods to stay aware of this literature for getting scientific knowledge of the latest research findings. The risks are most certainly not trivial, as decisions made on fallacious, answers may endanger trust or general well being and security of the public. But, with thousands of research papers being dispensed on the topic, making it more difficult to keep track of the latest research. Taking these challenges into account we have proposed COBERT: a retriever-reader dual algorithmic system that answers the complex queries by searching a document of 59K corona virus-related literature made accessible through the Coronavirus Open Research Dataset Challenge (CORD-19). The retriever is composed of a TF-IDF vectorizer capturing the top 500 documents with optimal scores. The reader which is pre-trained Bidirectional Encoder Representations from Transformers (BERT) on SQuAD 1.1 dev dataset built on top of the HuggingFace BERT transformers, refines the sentences from the filtered documents, which are then passed into ranker which compares the logits scores to produce a short answer, title of the paper and source article of extraction. The proposed DistilBERT version has outperformed previous pre-trained models obtaining an Exact Match(EM)/F1 score of 80.6/87.3 respectively.

Introduction
COVID-19 infections are characterized as a disease caused by a novel virus called Coronavirus now called extreme acute respiratory condition (SARS-CoV-2; previously known as 2019-nCoV) [1,2]. The worldwide reaction to COVID-19 has yielded a fast development of scientific publicationsexpanding at a pace of thousands every week -about COVID-19, SARS-CoV-2, different Covids, and related themes [3]. The people on the cutting edges of the battle -medical services experts, strategy makers, clinical scientists, and so on -will require expert specific methods to stay aware of this literature for getting proper knowledge of the latest research findings [4]. COVID-19 particular question-answering abilities are vital to making this abundance of information both helpful and significant. The risks are most certainly not trivial, as choices made on returned, fallacious, or counterfeit answers may endanger trust or general well being and secu-rity of the public. The author has recognized these dangers, however, accept the fact that our research technique benefits the more extensive COVID-19 research thus exceeding such dangers [5]. Taking these challenges into thought, our COBERT search system has the option to enormously help the endeavors of scientists to viably help to battle the current pandemic. The proposed system is a retriever-reader dual algorithmic approach that takes queries from the user as input and generates the most accurate responses consisting of a single line answer to the query, title of the literature, and a whole paragraph from the scientific literature [6][7][8].
The COBERT system searches a document of 59K corona virus-related literature made accessible through the Coronavirus Open Research Dataset Challenge (CORD-19) [9]. As of late, QA models have made huge advancements as far as both execution and throughput [10][11][12]. Such upgrades can be credited to the presentation of huge scope QA datasets and deep learning models, and the recent focus of the research towards the proficiency of such models. Lately, most of the QA models are extractors and re-rankers since their attention is on a generally small-sized dataset of text such as articles, sentences, or on the other hand entries, and so on [13,14]. Whereas, our framework works straightforwardly on a huge corpus of literature. Our proposed model coordinates best practices from the information retrieval with BERT to create a framework focusing on an end-to-end closed domain question answering system and analysis on a standard benchmark showing improvement over the past work. Our model fine-tunes pre-trained versions of BERT with SQaUAD adequately accomplishing high scores in recognizing answers [15] The lucidity of this configuration is one significant element of its architecture. Additionally, our strategy is explicitly customized for BERT-based models to use this architecture in the enormous scope of pre-prepared language modeling. [16][17][18] Our proposed model exhibits that the pretrained model diminishes the requirement for some intensely designed taskspecific models. BERT is the first fine tuning based model that accomplishes state-of-the-art results on a huge suite of sentence-level and token-level tasks, surpassing many task explicit models. Our proposed model holds the following components: Firstly, retrieval is performed throughout the whole corpus which is divided into paragraphs thus creating features based on tf-idf focusing on bigrams and unigrams. The embedding is then used to calculate the cosine similarity with the query and sequential comparison scores are obtained which are then used to Retrieve the top 500 documents with the optimal scores. Then, comparisons are performed batch-wise to get the document that is most probable to contain the answer.
The reader then does the extraction of the documents by further splitting them into sentences and these sentences are then Fine-tuned with Bidirectional Encoder Representations from Transformers (BERT) for refining the automatically generated answers based on the similarity between the query. We have also used DistilBERT (i.e., Distilled-BERT) for the task of fine-tuning.
Then comes our Ranker which ranks the most accurate answers using weighted scores threshold score obtained from the Retrieval and Reader. Ranker compares the scores of each candidate answer from each paragraph and compares the logits of the best answer for each paragraph. Then the best answer from that document is produced as output to the user.
This paper is organized in the 9 folds. Firstly, we will discuss the literature review thus analyzing the various past works in the domain in Sect. 2. Section 3 then focuses on our problem description briefly. Then, Sect. 4 explains our proposed methodology where each component of our architecture is described. In Sect. 5 we have explained our Dataset, Preprocessing, Training,etc. Section 6 reported the results and analysis of the comparison of various techniques and our model. Whereas, in Sect. 7 we described some Limitations of the study .We have concluded in 8 and discussed Future Works in Sect. 9.

Literature Review
Kricka L.J. et al. [8] concentrated towards shedding light on how AI is reshaping medical field and helping in COVID-19's diagnostic research through three prominent initiatives: COVID-19 focused datasets (e.g., ; AI-Boosted inspection means (e.g., WellAI, SciSight); and get in touch with tracing supported mobile communication technology. WellAI uses NLP neural networks to learn from CORD-19 so as to summarize the existing knowledge through unsupervised learning. SciSight is an AI-powered visualizing tool for scrutinizing associations between concepts appearing with CORD-19/ Contact Tracing is an extensive auditing process for clashing of a communicable disease through 3 steps: Identifying, Listing and Follow-up. Apps such as Aarogya Setu were released in Indian government to invigilate erstwhile COVID Survivors and mapping them to hatch contamination plans accordingly.
Broth Andreas et al. [10] proposed a Question-Answering (Q/A) system over Knowledge Bases(KB), which overwhelms impediments which were endured-namely, porting KBs language-agonistic systems(Eg. SemGraph QA), structural gap (Eg. AskNow, DEANNA), Scalability (SINA) . This technique is established on the perception that a lot of questions are often presumed from the connotations of the words within the question while the syntax of the question has subordinated consequence. A loss of credit that authors admit of their implementation is that the identification of relations relies on a dictionary. They have applied their exemplary algorithm and adapted a group of existing services in order that end-users can query, using multiple languages, multiple KBs at an equivalent time, employing a unified interface .
Yu Wenhao et al. [11] developed a TransTQA, which is a novel system that gives automatic responses by retrieving proper answers supported correctly answered similar questions within the past. TransTQA is configured with the ALBERT model consistent with the required arguments that defines the model architecture, producing transformer encoders. MLM technique is employed for fine tuning source corpus. The author has developed TransTQA, which may be a novel system that gives automatic response by retrieving proper answers supported correctly answered similar questions. TransTQA is made upon a Siamese ALBERT network, which enables it to reply to questions quickly and accurately. Furthermore, TransTQA adopted transfer learning to enhance its performance on multiple tech domain QA.
Abacha Ben Asma et al. [12] presented a Medical Visual Question Answering task (VQA-Med) focusing primarily on 4 categories of clinical questions : Modality, Plane, Organ System, and Abnormalities. The first three tasks are classified as : classification task and the fourth can be put with answer generation. All of these questions are solved through images only, and do not require any additional domain based context for generating answers, and were evaluated on Accuracy and BLEU score.
Nogueira Rodrigo et al. [13] re-implemented the query based passage re-ranking. Trained on state of the art TREC-CAR dataset, it beats out the previous top entry by 27% at MS MARCO passage retrieval task. The researcher's pipeline can be divided into 3 stages: first being retrieving all conceivable documents pertinent to the given query, using archetype operations such as BM25. The second stage implicates passage re-ranking by computationally-accelerated technique. The final stage being top ten or fifty of these documents being responses to the user's question. Using the BERT LARGE Model as binary classification model and [CLS] vector as input to a single-layer neural network to obtain the probability of being germane.
Gao Luyu et al. [14], had studied if and the way of knowledge for search within BERT itself, can be re stationed to a smaller ranker through distillation, producing up to ninefold speedup, while perpetuating state-of the art performance. The facility of a BERT ranker is from two main origins : 1) general purpose language modeling knowledge learned in pre-training, and 2) search-specific relevance modeling knowledge learned in fine-tuning.The researcher minutiates three enunciated methods that coalesce fine-tuning and distillation to reach at a smaller rank from an incipiently full-sized BERT model: 1. Ranker Distill: distillation is used for search knowledge, 2. LM Distill + Fine-tuning: distillation is serviced for LM knowledge, and 3. LM Distill + Ranker Distill: distillation is employed for both LM and search knowledge.
They found a higher degree of compression herald more predicaments in model optimization due to loss of knowledge.
Esteva et al. [19] implemented a retriever-ranker semantic search engine devised to handle complex queries regarding COVID-19. The retriever is composed of a Siamese-BERT (SBERT) as an encoder, with a TF-IDF vectorizer and reciprocal-rank fused with a BM25 vectorizer, as a ranking function to estimate the relevance of documents in a given search query. A query is given for which each document is provided with a retrieval score. The ranker comprises a multi-hop question-answering (QA) module that, alongside a multi-paragraph abstractive summarizer, adjusts retriever scores. The retrieved set of documents are run through a question answering module and an abstractive summarizer in the ranking. Based on this ranking, the documents are ranked by a weighted combination of their answer match, retrieval scores, and summarization match.
In [20] Abacha et al. approached one of the challenges in the large-scale information retrieval (IR) process that is developing accurate and domain-specific ways to answer natural language questions, especially when answering questions related to the medical domain. Dedicated Question Answering (QA) systems do not rely on any external information about the users, and hence, it makes them one of the viable solution approaches. Thus, the Recognizing Question Entailment (RQE) based QA system was developed to answer new medical questions using the existing answers to previously asked questions. A collection of 47K medical question answer-pairs was built and shared. This approach works well for both open-domain and specific-domain. Regardless, deep learning models achieved state-of-the-art results on an opendomain and clinical dataset, which showed low performance on consumer health questions.
Tang et al. [21] have created a dataset named COVID QA, containing 124 pairs of questions-article related to COVID-19. The architecture begins with keyword-based re-retrieval to determine a set of pertinent candidate documents that are then re-ranked by machine-learned models to assign higher relevancy documents to higher ranks. In the final stage, the model would input the query and the document as input and determine the most relevant passages. However, with limited data entries in the CovidQA dataset, it is impossible to train the QA model in a supervised manner. The dataset also lacks "no answer" examples, which is an unrealistic conjecture in real life.
In this paper, Lee et al. [22] proposed a system comprised of three folds: first, COVIDASK, a real-time QA on COVID-19; second, incorporation of traditional biomedical text mining tools into QA model to enhance usability (e.g., NEL); third, evaluation of COVIDASK on COVID-19 dataset created by the authors. Although COVIDASK is not an open-domain QA model, it has been made with many techniques from open-domain QA models since it has to handle a large amount of text. COVIDASK mainly focuses on reducing the latency in putting a query and receiving an answer. Latency can be reduced by decreasing the number of documents retrieved. Alternatively, all appropriate answer phrases are pre-indexed into dense and sparse vectors. Then, Maximum Inner Product Search is performed between query vectors and the phrase vectors to reduce the number of times a model passes through a document to only one.
The architecture proposed for the question-answering (QA) module, CAiRE-COVID by Su et al. [23], consists of three different modules: 1) Document Retriever: It preprocesses the user's query the most relevant n number of publications. It paraphrases long, complex queries to simple queries that are easier to comprehend by the system. These queries are run through the IR module, which returns paragraphs from the highest matching score. 2) Relevant Snipper Selector: The most relevant parts of paragraphs are highlighted and re-ranked based on the scores. 3) Multi-Document Summarizer: It returns an abstractive summary by summarising the top relevant paragraphs. This QA module is the perfect example of the extractor and re-ranked since their attention is on a small-sized dataset such as sentences or articles.
Many such models have been proposed to solve the problem of question-answering from a large corpus many of such have been focused on medical-question answering but the final capturing capabilities of models are proving ineffective when comes to a large corpus whereas our proposed model phase by phase increases the answering capability thus increasing the chances of the final answer being closer to ground answer as our model employs the retriever-reader dual algorithmic system. Whereas, coming to the deep learning models their performance is limited when capturing the embedding representations of question-answer pairs which generalize the model with one neural network which captures one-side features. Therefore it's necessary for the model to capture complex all-side features of QA-pair. Existing approaches deploys the heuristics on coming up with the possible answer whereas the proposed model is assessing its answering capabilities against the ground truth sentence in the ranker phase. The proposed architectures thus try to deal with such shortcomings which are discussed in the further sections.

Problem Description
In COVID-19 pandemic situation, many research analyses have been published in the form of a research paper, which is open access for all to do further analysis. Simultaneously, there were many solicitations that came from social orders (i.e., medical research community and Broader Society) for finding the response to the questions identified with Covid-19. Finding such sorts of answers is quite difficult with any system or can say no such system is capable of answering such kinds of questions. During the current situation of pandemic emergency, thousands of articles, blogs are being published, just some of which are empirical, factual, and peer-inspected. This may prompt the consideration of deceptive and the potential quick spread of scientifically reputable information or in any case fallacious exploration and information. Individuals on the forefronts -clinical Specialists, workers, and so on -are time-obliged in their capacity to parse this huge corpus, which could hinder their capacity to move toward the returned query items with the fitting degrees of distrust and queries accessible in less critical incidents [23]. To address this issue, we propose an end-to-end closed domain question answering system on COVID-19, which is present on the top of the HuggingFace BERT [24] transformers library. Since the system is an end-to-end cdQA system [17] that helps in answering a question related to the COVID-19 situation, mainly the questions proposed in the CORD-19 Kaggle. This proposed system based upon query, the user system gives the user three outputs, each having three components: the answer, Title of the paper, and Text described in the paper.

Closed Domain Question Answer (CDQA) Search
The CDQA based COVID-19 Search engine implemented here is based on a retriever-reader dual algorithmic approach, as shown in Fig. 1. The COBERT is model inspired by the DrQA Open Domain Question Answer model, developed by Chen et al. [25]. The main challenge with Question and Answering as a Natural Language Understanding task is that QA models often fail to perform when asked to produce an answer for a question from a large input text. To address the above-discussed challenge, the COBERT model was broken into two steps. Firstly, narrow down the input text to the top articles where the answer might be present (The Retriever) using search (e.g., TF-IDF, BM25), and secondly, it out of the narrowed down input text, find the best potential answer (the reader) using a QA model [4]. For this reason, ODQA and its CDQA are considered as an approach to providing "Machine Reading at Scale.
In other words, it can conclude that a closed-domain system [26] deals with questions under an exact domain, i.e., medicine or automotive maintenance. Table 1 shows a comparison of the ODQA and CDQA model. The domainspecific knowledge achieved by using this model fitted to a

Architecture
The COBERT pipeline architecture shown in Fig. 1 is made on the top of the Hugging Face transformers library. This system is classified into two different parts, such as Retriever and Reader, as shown in Figs. 2 and 3 respectively. Whenever a query is asked to the COBERT, as we can in Fig. 1 it is passed into the Retriever which selects k documents or articles from the corpus that are well on the way to contain the appropriate answer. Basically, it converts these documents into Tf-IDF vectorizer along with an input query and evaluates the cosine similarity of the document and input query. After selecting the most probable document, it passes these documents into the Reader. The reader is a pre-trained BERT model on our dataset. Now, the reader again calculates the best answers from the input documents. Finally, Ranker list down these candidate answer according to the logits score and threshold, and the final output is produced at the end.
The COBERT Retriever Pipeline schema shown in Fig. 2. The retriever schema chooses a couple of documents from a database to answer those questions that are set to a system. The COBERT system is working the same as a retriever of DrQA [27], which creates TF-IDF features based on uni grams and bi-grams. Further, this schema calculates the cosine similarity in-between the question sentence and every document of the database. TF-IDF Retriever segregated into two terms TF (Term Frequency) and IDF (Inverse Document Frequency). The TF is defined as t f (t, d), the least challeng-  ing decision is to exploit the term in the report using its raw count, i.e, the occasions that term t happens in record d shown in Equation 1 .
We have represented ft,d as raw count whereas tf scheme is The IDF, shown in Equation 2 is a proportion of the data the world gives, i.e., if it is normal or uncommon overall reports. It is the scaled logarithmically other division of the archives that comprise the word (acquired by separating, the all outnumber of reports by the number of records having the term, and afterward-logarithmic operation applied using the quotient): Where, N: total count of reports in the corpus. It is consequently normal to modify the denominator to 1+ d D: t d-. Furthermore, -d D: t d-is the number of reports when the term t shows up (i.e., t f (t, d) 0). If the term is not present inside the corpus, this will prompt a divisionby-zero. Then tf-idf calculated as shown in Equation 3.
Further, the system will divide the document into paragraphs by selecting the document. Next, it provides direction to the document reader with the question, which is a pretrained Deep Learning model. The proposed model used the BERT method (i.e., a Pytorch version of an NLP (Natural Language Processing) model) presented by HuggingFace. For our closed domain, we included the 59,000 articles as our corpus. As we have used the whole text of the paragraphs, this means we need to divide the data into chunks of 10,000 for pre-processing to avoid system crashes. The top articles identified based on a cosine similarity between the question string and the abstract text. Figure 3 shows that the reader used the most likely document, which is ranked by the retriever. To answer the question, the reader splits the document into a paragraph. Further, in the reader, a final layer is present in the architecture that matches with the help of an internal score function and the most frequent one based upon the scores is the final output. The reader is based upon Bert Processor using the 'distilbertbase-uncased' model. Section 4.6 explains the Distil BERT transformer model.

BERT
BERT, stands for Bidirectional Encoder Representation from Transformer the most recent refinement of a series of neural models that make substantial use of pretraining, and has prompted noteworthy gains in numerous natural language processing tasks, going from a text classification to do tasks like question answering from the corpus.
BERT eases the unidirectional requirement by utilizing a "masked language model" (MLM) pre-training objective. The MLM model arbitrarily masks a portion of the tokens from the information, and the goal is to foresee the masked word dependent on its neighbors (left and right of the word). Unlike directional models, which scans the text info successively (left-to-right or option-to-left), the MLM objective empowers the representation to utilize both the left and the correct setting, which permits to pre-train a deep Bidirectional Transformer. BERT advanced for several NLP problems like QA. It was introduced in 2019, generating language. That is why it does not require decoding. Figure 4 shows the original architecture of QA tasks using BERT, where the input sequence consists of two parts: the question asked and a special [SEP] token and then followed by the context [17,24]. It learns the context of a solitary word from the two words, which encompass it from both directions using its capability of Bidirectional mechanism [28]. BERTBASE and BERTLARGE were the two distinct architectures introduced in the BERT model. BERTBASE has a hidden size of 768, having a transformer block having size 12 and self-attention heads of 12 sizes. In the end, we obtain 110 million parameters.

Distil BERT
Distill* is a class of compressed models that started with DistilBERT (i.e., Distilled-BERT). It is a BERT based cheap, small, fast, and light Transformer model. Its performance is similar to BERT but runs faster by 60% but has 40% fewer parameters but still obtaining 97% performance if we compare it to BERT. These results were obtained using the GLUE language understanding benchmark. While using DistilBERT, it carries many similarities in comparison to the original BERT model. Thus, it (i.e., DistilBERT) is an interesting option to put a large-scaled trained Transformer model into production. The system has used a distilbert-BERT English language model, pre-trained on the same data used to pre-trained BERT (concatenation of the Toronto Book Corpus and full English Wikipedia) using distillation with the supervision of the Bert-base-uncased version of Bert. The model has 6 layers, 768 dimensions, and 12 heads, totalizing 66M parameters.

Ranker
Once the retriever and reader models are used, our solution presents the top 3 answers based on a weighted score between the retriever score (based on TF-IDF cosine similarity explained in Sect. 4.2) and reader score (based on DistilBERT QA Q-A pair probability). Cosine similarity (cos( − → q , − → d )), as shown in Equation 4, helps to find the similarity among two vectors of inner product space and conclude whether these vectors are indicating in the same direction. We frequently calculate the similarity of documents in tasks such as text analysis.
Where, [q i ] th term tf-idf weight in query both are length-1 vectors which are normalized.

Dataset
COBERT system uses the dataset of CORD-19: COVID Open Research Data set collected by The White House, with the help of leading research groups like Allen Institute for AI and Kaggle [26]. The dataset consists of a collection of work of several Researchers and its analysis. It consists of around 59 thousand papers and around 41 thousand full texts [27] incorporating papers distributed in more than 3200 journals. Many texts are related to institutions situated in the United States (over 16 thousand papers) followed by the United Kingdom (over 3 thousand papers) and the European Union and then Asian countries. Chinese organizations have seen a brilliant ascent this year (over 5K papers) because of China's status as the principal focal point of the COVID-19 episode, thus, making the source of data very much diverse.

Pre-processing
The files stored in the JSON format and processed in the form of chunks of size 10,000 one by one and the final output was generated in the form of .csv format having attributes like 'title,' 'abstract,' 'paragraphs,' where 'paragraphs' specifies the complete text of the specific research paper. The data then translated using Google Translator into English, as our main aim is to provide a QA system in the English language. Now the paragraphs column further split into sentences to optimize the training of the model. We have used the Dis-tilBERT model for the training. DistilBERT model trained on SQuAD 1.1 using Knowledge Distillation and Bert-baseuncased-whole-word-masking-finetuned-squad as a teacher. A separate copy is prepared using json format according to the SQuAD1.1 dataset for evaluation of the model.

Hardware and Training
We have used an Nvidia Tesla P100 GPU for training running on CUDA 9.2. 148 and cuDNN 7.4.1, having 16.4 GB RAM. The model trained using a pre-trained model of DistilBERT Tuning batches of size 400 in a chunk of 9000 rows at a time having a maximum sequence length of 384. Trained on GPU for 10,000 steps. Table 2 shows the hyper parameters of both of the pretrained weights namely BERT Base Uncased and Distil BERT with Knowledge Distillation fine tuned on CORD-19 dataset using Reader.
We have set the total batch size for training to be 128 and 256 in both the models. 'Max_query_length' are the size of the input tokens of the question, if found longer than this would be truncated. 'Doc_stride' is the size of stride when splitting of documents into chunks is performed. 'Max_seq_length' is the size of the input document sequence length which is set to 384. Whereas, 'max_answer_length' denotes the maximum size of the answer that can be generated which is set to 30. Adam weight decay was used as an optimization having a learning rate of 1e-8 finally. All the other hyper parameters are set to default In addition, fine-tuning performed on the reader using an annotated dataset having the same format as the SQuAD dataset. The model needed two attributes, namely the 'title' and 'paragraphs.'

Prediction and Evaluation
A list of queries was prepared for the initial testing phase consisting of question like: -"What is the optimal quarantine period for coronavirus COVID-19", -"What is a range of incubation period for coronavirus SARS-CoV-2 COVID-19 in humans", -"What is an effective quarantine period for coronavirus COVID-19"?
During each batch of retriever fitting, predictions of the queries simultaneously are done and stored in the form of a data frame. Later on, the answer with the best score used to produce the output. We have kept the retriever score as 0.35 for the predictions that presented the best weight on the development set of SQuAD 1.1-open. Each of the question predictions were stored in a separate data frame have columns such as: "answer","probability","start","end","qas_id","title", "paragraph","retriver_score","final_score". For getting the final results the data frame is sorted upon "final_score" to obtain the 15 best predictions. For evaluating the model, it requires three steps. Firstly, the panda's data frame converted into JSON format with SQuAD version 1.1 formats. Then, using a cdQA-annotator, we prepared an annotated JSON format file to add ground truth question-answer pairs for evaluation of the model. The third and final step is to evaluate both the JSON file prepared in the above steps to get the Exact Match (EM) and F1 score.

Metrics
To measure the accuracy in problems such as QA, EM (Exact Match), and F1 scores are regularly used. Exact Match: This metric is one of the most widely used metrics for evaluating the results of question-answering tasks. For each pair of question and answer if the words in the predicted answers are exactly matched with the ground truth answer then EM score 1 otherwise 0. It is straightforward as it sounds. F1 score calculates the similarity to the ground truth. F1 is harmonic mean among recall and precision, as shown in Equation 5, 6, and 7, respectively. F1 score measured using the following criteria's: 6 Results And Comparison Analysis Figure 5 shows an example of the two queries when given as input to our model. The system outputs not only an answer but also the title of the document/article, the paragraph where the answer was found. In addition, every query model outputs three answers. For the sake of simplicity, we have presented only a single instance of the answer to the query. This can be set to multiple answers also. We observed that the model outputs more accurate answers in the query containing frequent terms related to coronavirus. Apart from score evaluation, through the use of Metrics described in Sect. 5.5, multiple such queries were checked on by multiple users for a better evaluation by the author. And both the results were satisfying according to the feature generation capabilities of the COBERT system. In the screenshot, we can see the answers generation quality that COBERT is handling, this capability is likely to increase along with the content of literature given to our system. The model comparison based on EM and F1 score shown in Table 3, Figs. 6, and 7. GPU version of BERT (with sklearn wrapper) is a version of the BERT model trained on SQuAD 1.1 runnable on GPU. It is available only with a sklearn wrapper achieving an EM score of 81.2% and 88.6%, whereas after fitting the pipeline on the CORD-19 corpus, the model achieves 79.3% EM and 86.4% F1-score. BERT for the QA model is CPU/GPU agnostic. The model was loaded in a machine with support for CUDA. It automatically sends the model to GPU for computations. This  Here there is an increase in EM, and the F1 score is comparable. DistilBERT model trained on SQuAD 1.1 using Knowledge Distillation and Bert-largeuncased-whole-word-masking-finetuned-squad as a teacher. This version of DistilBERT achieves 80.1% EM and 87.5% F1 while being much faster and lighter. Whereas, after fitting the pipeline on the CORD-19 corpus, the model achieves 81.6% EM and 87.3% F1-score. Here, we achieved EM highest among all the pre-trained models on SQuAD 1.1 dev and fitted model of CORD-19 as well, whereas achieved a comparable F1 score. Although both the DistilBERT's EM score improved on the fine-tuned data, due to large size input document size of the CORD-10 dataset as compared to SQuAD1.1 but still significant increase in score of 0.4 shows that Distil-BERT performs better on large input sequence size dataset. Whereas, it's not the case with the BERT uncased version which saw a decline in EM in GRU version but still improved in non-GPU version. Therefore, we can say that our Distil BERT for QA with Knowledge Distillation after considering all the above models performs better than BERT on data with large input sequence with 60 per cent faster performance in terms of speed and 40 percent smaller size than BERT-uncased whose reason is due to its triple loss combining language modeling.

Limitations
As the system is developed on literature based dataset of CORD-19 hence the answerability of the model on unstructured dataset is still a domain of research which is not incorporated in the current study. Also, we attempted to set a limit for the scores of answers, the conveyance of scores were very conflicting across various queries. The proposed model captures the context-sensitive features only from the generated embedding.Our system is pre-trained on a closed domain of COVID-19 literature that too in English language only. There is also a scope to integrate the system with various trusted websites and articles too.

Conclusion
In this paper, COBERT-question answering system architecture was proposed where our outline was to tackle challenges of COVID-19 to aid researchers and clinical workers to get the authentic scientific information at ease. This model was based on a three component architecture of Retrieval, Reader and Ranker to automatically answer .We show that our architecture is able to capture similarity among query and large number of text documents .Due to the portrayed attributes of our methodology portability is guaranteed which is a critical favorable position in correlation to past methodologies. We have indicated the enormity of our methodology in a broad assessment over various benchmarks. We have finetuned the BERT uncased and DistilBERT version of BERT in the Reader phase on the CORD-19 dataset then compared documents on the basis of cosine similarity to get the best answer. The results of the system were tested on a diverse set of questions and found to be consistent by the users. We have shown with help of empirical results that DistilBERT is able to capture features dependencies of long documents in a better way due to its triple loss combining language modeling obtaining a EM/F1 score of 81.6/87.3. Our proposed model is additionally simple to be summed up to general area and has the option to enormously help the endeavors of scientists to viably help to battle the current pandemic.

Future Works
We would like to consider creating knowledge based graphical representation to map the similarity of extracted documents to integrate them on the basis of ranking and referral perspective. To incorporate unstructured text an extra component called verifier can be included which tactfully extracts the answer collectively from the unstructured data too. To capture not only context-sensitive features but also semantics which are structured which can provide rich language representation using techniques such as role labeling before feeding the input sequence to BERT then using self-attention to capture word-level representation. Also, we want to explore the multilingual capabilities of the model along with adding real-time content from resources such as trusted websites.
Funding This research recieved no external funding.

Conflict of interest
The authors declare that they have no conflict of interest.
Code availability We can upload as per requirement.


Cost Effective Hybrid Fault Tolerant Scheduling Model for Cloud Computing Environment

Cloud computing provides flexible and cost effective way for end users to access data from multiplatform environment. Despite of support by the features of cloud computing, there are also chances of resource failure. Hence there is a need of a fault tolerant mechanism to achieve undisrupted performance of cloud services. The task reallocation and duplication are the two commonly used fault tolerant mechanisms. But task replication method results in huge storage and computational overhead, when the number of tasks is increasing gradually. If the number of faults is high, it incurs more storage overhead and time complexity based on task criticality. In order to solve these issues, we propose to develop a Cost Effective Hybrid Fault Tolerant Scheduling (CEHFTS) Model for cloud computing. In this model, the Failure Occurrence Probability (FoP) of each VM is estimated by finding the previous failures and successful executions. Then an adaptive fault recovery timer is maintained during a fault, which is adjusted based on the type of faults. Experimental results have shown that CEHFTS model achieves 43% reduced storage cost and 13% reduced response delay for critical tasks, when compared to existing technique. Keywords—Cloud computing; failures; fault tolerant; critical tasks; scheduling; fault recovery; overhead

I. INTRODUCTION
Cloud computing is one of the new areas of technology that has emerged recently. On-demand resource sharing can be accomplished by utilizing internet and cloud services [1]. The primary goal is to offer services or distribute resources at the request of the customer with the lowest possible cost. There is a huge amount of heterogeneous resources [2], a vast user base, and a great range of application tasks in the cloud computing system. They have to deal with numerous individual activities, as well as big data. In terms of finance and science, cloud computing is a very valuable service [3]. A difficult challenge arises when deadlines are imposed on the jobs, since the required resources may be unavailable due to failures [4].
Along with missing project deadlines, fault tolerance is an important cloud computing aspect. It is designed to handle real-time tasks even if a failure occurs. Due to the higher storage overhead of taking backups, it is vital to use alternative approaches that have a high resource utilization ratio. Even though cloud features are attractive and a fault tolerant technique is required to guarantee uninterrupted cloud service functioning [5].
Both internal and external factors contribute to some of the flaws. The two most often utilized fault-tolerant strategies in cloud computing are task reallocation and duplication. In task reallocation, a task is reassigned after a fault occurs. This method boosts the overall system resource usage. However, failing to meet the deadline constraints [6] of jobs could cause the response time to be prolonged. The rise in the amount of electricity consumed while making resource allocations causes resource requests to fail in data centers.
Both traditional application components and new traditional application components may be heterogeneous in their component composition, composed of scientific techniques. With virtualization technology, these components have their own specialized execution environment. Workflows can also be implemented and run in cloud environments that offer an ever-growing resource pool [7] that you pay just for what you use. On-demand cloud resources can thus be acquired and shared on workflows. Clouds serve as a favored scientific process execution environment due to these characteristics.
However, processes that operate in the clouds face obstacles, because the complexity and dynamics of processes and cloud breakdowns occur more frequently. Failures tend to stop continuous implementation and have considerable impacts on the performance of processes, particularly for big, long-run workflows. It is crucial that scientific workflow scheduling fault tolerants be achieved in order to provide consumers with flawless experience.
Due to the time-consciousness of many scientific procedures, their successful completion does not depend exclusively upon the correct results but also on the time when those results are available. For a workflow that must complete the working phase and generate the computer results, a deadline is specified for the workflow. Failures lead to deadline conscious workflows that cannot be concluded on time without an appropriate faults-tolerant schedule. In that circumstance, QoS is badly damaged, although beyond the deadline [8] the results may be acquired. Therefore, it is required and crucial to plan a defect tolerant workflow since processes can be successfully completed before the deadline. 416 | P a g e www.ijacsa.thesai.org This paper is organized as follows. Section 2 presents the motivation and objectives. Section 3 presents the proposed methodology. Section 4 presents the simulation results. Section 5 presents the conclusion.

II. MOTIVATION AND OBJECTIVES
In fault-tolerant workflow scheduling [6], task replication method is used. Here duplicate multiple copies of tasks [9] can execute simultaneously. But it results in huge storage and computational overhead, when the number of tasks is growing. Moreover, it did not present the strategy for finding the exact number of duplicate copies.
The Power, memory, and other network elements were addressed in order to enhance the reliability of resources. In task scheduling, the VM with the greatest level of reliability is picked. Hussein El Ghor et al. [8] has presented two scheduling algorithms. The first approach uses energyefficient fault-free scheduling and the second approach provides essential slack time for fault recovery. The proposed model is limited in handling small group of tasks that cannot improve the performance levels.
Redundancy is a common way to protect against errors that can be done quickly or in space. Any additional resources required to perform the identical duplication or resilience replication process are designated as spatial redundancy. When considered in depth, an example of spatial redundancy may be parallel execution. There are considerable space expenses when you simultaneously run many jobs on different resources. In the case of redundancy, the task that was missed with respect to initial resources does not have to be redone because the clock was reset.
With Fault Tolerance Algorithm utilizing Selective Mirrored Tasks Approach (FAUSIT), they proposed a selective mirrored task method to ensure the correct balance between parallelism and topology for applications. DAGbased application fault tolerance is dealt with by limiting the number of make spans and minimizing computation costs.
For important jobs or tasks with permanent failures, the dynamic fault-tolerant workflow scheduling [10] uses spatial re-execution (SRE) and temporal re-execution (TRE) schemes in unison. However, as the number of issues detected for important operations is greater, the SRE strategy consumes more disc space. If the number of faults for non-critical tasks is excessive, then it has an adverse effect on time complexity. Additionally, the backups' odds of failure are not examined.
In FESTAL [11], for each primary task, a backup task is created and VMs are allocated for both. When the primary task is completed successfully within the execution time, then the execution of back task is terminated. But in this approach also, it results in huge storage and computational overhead [12], when the number of tasks becomes high. The performance levels are not satisfactory as the model causes overhead in the system.
In [13], a fault tolerant technique to ensure task completion deadline, has been proposed. But it does not check the types of tasks and storage overhead also is more that improves the delay levels in the system.
In [15], a trust based scheduling technique is proposed. For selecting the trusted computation service, set-based particle swarm optimization (S-PSO) and for selecting the storage service, ser covering problem (SCP) tree search are applied. The task scheduling concept in [16] is built on trust mechanisms. This model uses the Bayesian cognitive technique to determine the trustworthiness of computer nodes by examining the trust relationships that exist between them.
A research paper [17] outlines a trust-based Meta heuristic workflow scheduling technique called TMWS. The manual offers strategies and procedures to avoid the hazards associated with workflow scheduling.
Hence the fault tolerant algorithm should meet the following objectives: • To reduce storage overhead.
• To reduce time complexity.
• To reduce the chances of backup VM failures.
• To meet deadline of critical tasks.
• To reduce the response delay.
• To reduce the fault recovery time.
• To increase the CPU utilization.
In order to solve these issues, we propose to develop a cost effective hybrid fault tolerant scheduling scheme for cloud computing.

A. Overview
In this paper, Cost-effective Hybrid Fault Tolerant Scheduling (CHFTS) model is proposed. Figure 1 shows the block diagram of the CHFTS model.
The CHFTS model uses previous failures and successful executions to calculate the VM failure occurrence probability (FoP). The anticipated execution time (EET) is estimated for each job Ti. At this point, the projects with deadlines of less than one week are classed as high-priority activities. The designation of some VMs as fault-tolerant based on FoP. A fixed number of primary and backup VMs are assigned to each task. A recovery timer is initiated if a failure occurs in any VM. A notification will be delivered to the failed VM, and the VM will continue execution from the point in time that the failure occurred.

B. System Model
Virtualization divides each server in the Cloud system into a series of heterogeneous virtual machines (VMs). The fact is that a VM is an essential component of a cloud system.
Suppose that a cloud system offers a set of VM resources, as given by.  The model represents the system structure, as seen in Figure 2. Processing capacity and cost per hour are particularly important for selecting VM(k) because of processing capacity and cost per hour, respectively (k). Users can access an endless number of VMs on cloud computing platforms that utilize virtualization. In addition, all VMs are hosted in one cloud data center to provide consistent bandwidth for all the VMs.

C. Fault Model
If a task is incomplete because of internal or external issues, it may be considered as a task failure.
Faults may occur during the execution of scientific work flows in cloud computing environments. There are various reasons behind these faults. The most common reason for task failure is the failure of corresponding VM. The other reasons for faults include unavailability of enough resources, resource overloading, delayed execution time etc.
There are two cases of faults that may occur in cloud, during execution. The first one is a permanent fault and other one is a temporary fault. In temporary faults, the faults may be recovered within short span of time whereas in permanent faults, the faults cannot be rectified until the failed element is repaired or replaced. Fail-signal or acceptance test is a fault detection mechanism, commonly used. Due to task failures, the make span time of work flows will be increased and the service-level agreement (SLA) will be violated. Re-execution is one of the commonly used less expensive fault tolerant technique, which improves the reliability of workflows. Re-execution can be implemented in two methods: Spatial re-execution on other resources (SRE) and Temporal re-execution on the same resources, after fault recovery (TRE).

1) Estimation of FoP:
The exponential probability density function for a specified mean time (M) between failures, is given by.
Then, the probability of a failures, that occurs before time Δ for a VM is given by.
Hence the probability of successful completion during the time Δ, is given by.
Let be the computation time for a subtask executing on a VM and be the computation interval between two check points.
Then, the average number of attempts ( A) required to finish is given by.
The number of success is given by Hence the number of failures NoF, during Δ is given by Let S(t i ) and W(t i ) be the workload size of given input task t i .
If and only if all the input data has been received from all prior tasks, then the task is started (ti). start time of t i is signified by Eq. (9).
Note that if t i =t enrty , then T start (t i )= 0. Then, the transmission time is computed by, where BW is the bandwidth between two VMs.
Then, the execution time is given by, Thus, the end time of task t i is given by.
Hence the make span of workflow is given by. For each task, a pair of VMs <VM p , VM b > from the k VMs, are allocated, where VM p and VM b are primary and backup VMs. If there is a fault occurs in VM p , then a fault recovery timer (T FR ) will be started.
Case-1: If the task is a critical task, then T FR is fixed such that.
T FR < EET-FOT, where FOT is the time of fault occurrence.
Case-2: If the task is a non-critical task, then T FR is fixed such that.
If the fault cannot be recovered within T FR , (ie) If fault recovery time (FRT) is more than T FR , then immediately, a notification will be sent to VM b , which will resume the execution from that point.

Algorithm -Fault Tolerant Scheduling
Input : List of VMs and Tasks Output: fault tolerant scheduling 1. foreach VMj 2.
End if 6. End for 7. foreach task t i 8.

end For
Special outcomes of this approach 1) Since fault free VMs are selected, the chances of faults are less.
2) It does not result in huge storage since the VMs are allocated only when the fault is not recoverable (permanent fault) within a time span.
3) The time critical tasks will be executed within their deadlines since the tasks are migrated to backup VM, within the deadline.

4)
The non critical tasks can be executed in the same VMs if the fault is recovered within tolerable time span (which is an even multiple of its deadline). 5) If the task could not be recovered beyond the tolerable time span, then only the non critical task are migrated to backup VMs, thereby reducing the storage overhead significantly.
6) Since the execution of tasks starts in the backup VM only from the time of fault, the time complexity will be less.

IV. EXPERIMENTAL RESULTS
The proposed CEHFTS model is compared with the Spatial-Temporal Re-Execution method (SRE) [10]. The NASA workload [14] has been used as the emulator of Web users requests to the Access Point (AP). This workload represents realistic load deviations over a period time. It comprises 100960 user requests sent to the Web servers during a day. Table 1 shows the experimental parameters assigned in this work.

A. Results for Critical Tasks
In this section, work flows with strict deadlines are considered (critical tasks) and allocated to VMs. The FoP of each task is increased in terms of fault rate. The performance metrics response delay, percentage of missed deadlines, CPU utilization and storage overhead of server are considered for evaluation. Table 2 Performance Results of CEHFTS and SRE schemes for fault rate of critical tasks. Figure 3 shows the response delay measured for the critical tasks when the fault rate is increased from 1 to 2. Since in the case of time bound critical tasks, backup VM migration is mostly supported, the response delay will be high. Hence, CEHFTS has the delay in the range of 1.4 to 2.8 seconds whereas SRE has delay in the range of 1.7 to 3.1 seconds. However, in case of CEHFTS, the failed tasks are resumed when the recovery time is less than EET. Hence it has 13% reduced response delay, when compared to SRE scheme.  Figure 4 shows the missed deadlines for the critical tasks when the fault rate is increased from 1 to 2. Since in the case of time bound critical tasks, if the recovery time is higher than EET or if the fault is permanent fault, the chances of deadline miss are high. As depicted from the figure, CEHFTS has the missed deadlines in the range of 18% to 41% whereas SRE has missed deadlines in the range of 25% to 47%. However, in case of CEHFTS, the failed tasks are resumed when the recovery time is less than EET. Hence it has 19% reduced missed deadlines, when compared to SRE scheme.    Figure 5 shows the CPU utilization for the critical tasks when the fault rate is increased from 1 to 2. In case of permanent failures, the CPU utilization of primary VMs will be affected for the critical tasks. As it can be seen, from the figure, the utilization of CEHFTS reduces from 57% to 41% and the utilization of SRE reduces from 47% to 33%. However, CEHFTS has 18% higher CPU utilization than SRE scheme, since it minimizes the use of backup VMs.  Figure 6 shows the storage cost of servers in case of critical tasks. Since in case of time bound critical tasks, if the recovery time is higher than EET or if the fault is permanent fault, the tasks are migrated to backup VMs, leading to increased storage cost. As depicted from the figure, CEHFTS has cost in the range of 150 to 470 whereas SRE has cost in the range of 270 to 840. However, in case of CEHFTS, the failed critical tasks are resumed when the recovery time is less than EET. Hence it has 43% reduced storage cost, when compared to SRE scheme.

B. Results for Non-Critical Tasks
In this section, work flows with elastic (flexible) deadlines are considered (critical tasks) and allocated to VMs. The FoP of each task is increased in terms of fault rate. The performance metrics response delay, CPU utilization and storage overhead of server are considered for evaluation.   Figure 7 shows the response delay measured for the noncritical tasks when the fault rate is increased from 1 to 2. Since in case of non-critical tasks, the fault response timer is set as high, the response delay will be higher than that of critical tasks. Hence, CEHFTS has the delay in the range of 2.6 to 6.4 seconds whereas SRE has delay in the range of 3.5 to 8.3 seconds. However, in case of CEHFTS, the failed tasks are resumed when the recovery time is less than EET. Hence it has 25% reduced response delay, when compared to SRE scheme. Figure 8 shows the CPU utilization for the non-critical tasks when the fault rate is increased from 1 to 2. In case of permanent failures, the CPU utilization of primary VMs will be affected for the non-critical tasks also. As it can be seen, from the figure, the utilization of CEHFTS reduces from 52% to 40% and the utilization of SRE reduces from 45% to 30%. However, CEHFTS has 16% higher CPU utilization than SRE scheme, since it minimizes the use of backup VMs. Figure 9 shows the storage cost of servers in case of noncritical tasks. Since, if the fault is permanent fault, the tasks are migrated to backup VMs, leading to increased storage overhead. As depicted from the figure, CEHFTS has cost in the range of 120 to 270 whereas SRE has cost in the range of 165 to 472. However, in case of CEHFTS, the failed noncritical tasks are resumed when the recovery time is less than EET. Hence it has 35% reduced storage cost, when compared to SRE scheme.

V. CONCLUSION
In this research, a new hybrid fault tolerant scheduling approach, called Cost Effective Hybrid Fault Tolerant Scheduling (CEHFTS), is proposed for cloud computing. For this model, the prior failures and successful executions are used to estimate the VM failure probability (FoP). Estimated anticipated execution time (EET) is calculated for each task. At this point, the projects with deadlines of less than one week are classed as high-priority activities. An adaptive fault recovery timer is kept running until a fault occurs, which is then increased as a function of the various fault types. Experiments are conducted for critical and non-critical tasks by varying the fault rate. Experimental results have shown that CEHFTS model achieves 43% reduced storage cost and 13% response delay for critical tasks, when compared to existing technique. Future work focus on grouping the tasks based on its type and requirements so that the CPU utilization and battery power can be further reduced. The task deadline and time scheduling can still be improved to enhance the performance levels.


Memorable Maps: A Framework for Re-defining Places in Visual Place Recognition

This paper presents a cognition-inspired agnostic framework for building a map for Visual Place Recognition. This framework draws inspiration from human-memorability, utilizes the traditional image entropy concept and computes the static content in an image; thereby presenting a tri-folded criterion to assess the 'memorability' of an image for visual place recognition. A dataset namely 'ESSEX3IN1' is created, composed of highly confusing images from indoor, outdoor and natural scenes for analysis. When used in conjunction with state-of-the-art visual place recognition methods, the proposed framework provides significant performance boost to these techniques, as evidenced by results on ESSEX3IN1 and other public datasets.

I. INTRODUCTION
Visual Place Recognition (VPR) is a well-defined but highly challenging module of a Visual-SLAM (Simultaneous Localization and Mapping) based autonomous system [1]. It represents the ability of a robot to 'remember' a previously visited place in the world map and thus subsequently generating a belief about the robot's location in the world. VPR is mostly used in combination with metric SLAM techniques to perform loop closure [2]. Recent advances in SLAM research can be broken down into semantic mapping (surveyed in [3]) and Visual Place Recognition (surveyed in [1]), where the latter can be annexed into the former [3]. This specific work concerns metric, topological and topometric maps having single/multiple images as nodes (landmarks) of the map.
Traditionally, for visual place recognition, 'Places' have been selected/sampled based on time-interval [4], distance [5] or distinctiveness [6] in different approaches. These approaches are discussed in depth in the next section. Most of these methods attempt to reduce the size of robot's map and do not quantify if a sampled/sub-sampled image is a good representation of a place; thereby has a greater chance of matching upon revisiting. The quality of image selection mechanism restricts the performance of a VPR system, both in the short-term and long-term. Due to limited number of images being stored in the map, it is critical to select those images that can be matched successfully upon repeated traversal-the motivation for this research.
In this work, we look at image selection from a semantic point-of-view and draw inspiration from images memorable Fig. 1. A block-level overview of the proposed memorable-maps framework. Each image is evaluated for its memorability, staticity and entropy before insertion into 'memorable-map'.
to a human-cognition system. We use a Convolutional Neural Network [7] to compute the memorability of an incoming camera frame. However, while objects like cars, pedestrians and vehicles in an image are subjectively-memorable; they are intrinsically not good for place recognition as such dynamic objects are rarely re-observed. We thus perform object detection to compute the staticity of an image and mask memorability of dynamic content. In addition to being memorable and static, an image has to be content-rich for which we calculate the entropy map.
The contribution of this work is a semantically coherent framework ( Fig. 1) that filters an input image through a tri-folded criterion. Hence, ensuring that every image to be inserted against a place in robot's map is a good representation of the said place and highly recognizable. To analyze the effectiveness of this framework, we created a dataset 'ESSEX3IN1' from indoor, outdoor and natural environments. Unlike existing VPR datasets, ESSEX3IN1 mimics a robot exploring an environment instead of traditional path-following and is thus composed of highly confusing images from all three environments. We show how these confusing images lead to poor performance of current visual place recognition systems. The final results show the effectiveness of proposed framework in segregating these 'confusing' images from 'good' images, thereby increasing VPR precision and reducing database size. We also evaluate our framework on other public VPR datasets to show that this performance enhancement can be generalized.
Due to its agnostic nature, any VPR technique can obtain a performance boost by stacking the presented framework as an additional layer in the VPR pipeline.
The remainder of the paper is organized as follows. In Section II, a comprehensive literature review regarding Visual Place Recognition state-of-the-art is presented with focus on image selection and semantic-mapping. Section III presents the motivation, design and implementation details of the 'memorable-maps' framework developed in this work. Section IV is dedicated to the experimental setup for evaluating and analyzing state-of-the-art VPR techniques with and without proposed framework. Following-up on Section IV, Section V puts forth the results/analysis obtained by combining memorablemaps and contemporary VPR techniques on multiple public datasets and ESSEX3IN1. Finally, conclusions and future directions are presented in Section VI.

II. RELATED WORK
VPR and SLAM have seen major developments through different cognitive, intuitive or semantic approaches to the problem. A comprehensive review of these techniques is performed by Lowry et al. [1]. An earlier work on probablistic implementaion of SLAM in visual-appearance domain, called 'FAB-MAP', is presented by Cummins et al. [8]. This work was combined in [9] with a biologically inspired SLAM technique 'RAT-SLAM' [10], mimicking the Rat's hippocampus. Milford et al. [11] utilize sequence of images instead of individual frames to successfully match previously visited places under significant environmental variations. Similar to other fields, Convolutional Neural Networks (CNNs) have been a game-changer for VPR. The application of CNN for VPR was first studied by Chen et al. [12]. Authors in [13] trained two dedicated Neural Networks for VPR on Specific Places Dataset (SPED) containing images from different seasons and times of day. Unlike previous implementations where image descriptors were manually formed from CNN layer activations, Arandjelovic et al. [14] trained a new VLAD (Vector of Locally Aggregated Descriptors) layer for an end-to-end CNN-based-VPR. For images containing repetitive structures, Torii et al. [15] proposed a robust mechanism for collecting visual words into descriptors. Synthetic views are utilized for enhanced illumination invariant VPR in [16], which shows that highly condition variant images can still be matched if they are from the same viewpoint. In [17], authors try to extract local features from convolutional layers corresponding to salient Regions of Interest (ROI), thus providing significant viewpoint invariance. State-of-the-art performance is shown by authors in [18], by combining VLAD descriptors with ROI-extraction to show immunity to appearance and viewpoint variation.
Traditionally, places have been described by camera frames, where a place is selected from multiple video frames based on either time-step, distance or distinctiveness. Most [24] are timebased, as frames are selected given a fixed FPS (frames per second) rate of a video camera. However, time-based place selection assumes a constant non-zero speed of the robotic platform and is thus impractical in many situations. To cater for variable speed, distance-based frame selection is used where a frame is picked every few metres to represent a new place [25] [5]. Both time-and distance-based approaches lead to huge database sizes and frequently sample visually identical frames as different places; thus leading to inaccuracies and impracticality for long-term autonomy.
Different research works have tried to overcome these intrinsic limitations of image sampling by proposing image selection based on visual distinctiveness. Chapoulie et al. [6] use a customised algorithm that detects change point for segmentation between different topological places in both indoor and outdoor scenes. Image sequence partitioning for creating sparse topological maps is presented by Korrapati et al. [26], where sequences of images are divided into nodes/places using four descriptors namely GIST, Optical Flow, Local Feature Mapping and Common-Important Words. In [27], a thematic approach is adapted to evaluate the novelty of an incoming image by co-relating it with the redundancy of visual features/topics. Bayesian surprise is adapted with immunity to sensor type, for extracting landmarks to create a sparse topological map in [28]. Online topic modeling with visual surprise calculation is done by Girdhar et al. [29] for under-water explorations. An incremental unsupervised place discovery scheme is adopted by Murphy et al. [2] which fuses information over time to find visually distinct places.
Semantic mapping techniques for summarizing a robot's experience are surveyed by Kostavelis et al. [3]. Authors in [30] present both offline and online solutions for finding images that best summarize a given sequence. The score for every incoming image is related to the difference of posterior distribution from prior distribution using bayesian surprise or set theoretic surprise. In [31], coresets are used to pre-cluster input image stream and then topic-based image representation is used followed with graph-based incremental clustering. A place detection scheme is proposed by Karaoguz et al. [32] based on bubble-space representation. A new place is checked for informativeness based on surface deformation and variance in a time-window of coherent images. The authors in [33] use region proposals in spatio-temporal context instead of low-level features to represent input frames and then based on region-adjacency-graph detect visually distinct places. A human-augmented change point detection scheme is presented by Topp et al. [34] where a change stimuli could either be pointed out by the robot or its operator. The authors propose the change as a structural ambiguity, which can be pointed out either by the robot or a human operator during a guided tour. Detection of change point is also targeted by Ranganathan [35] with a Bayesian probablistic model. One common element to all these works is that they focus on map compression, video segmentation or experience summarization, but do not discuss if the resulting compressed/summarized map is actually composed of good matchable images of places. These methods define the distinctive nature of images based on their visual difference from previously seen images. Resultingly, such visually different images may come from grassy plains, natural scenery, dynamic objects or low-textured places leading to poor VPR performance. Drawing inspiration from the said, we in this work, define distinctiveness based on a place's mem-orability (cognitive), static-content and information-richness leading to highly matchable compressed maps dubbed as 'memorable-maps'.
In addition to the above, we discuss two works having similar motivation to our approach. The interesting work by Hartmann et al. [36] proposes a random forest classifier of 5 decision trees trained on a dataset of 455 outdoor images. The objective of this random forest is to find keypoints in an image with low matchability and subsequently discarding them. This technique is computationally intensive in comparison to our methodology as we compute a single matchability (memorability) score against an image instead of scores against each image keypoint. Moreover, in VPR, features coming from dynamic objects and low-textured scenes are usually not re-observable/matchable (as shown later in our paper) but have not been examined in [36]. Although vegetation is considered to belong to non-matchable category, results show features coming from trees as being classified as matchable in [36]; which usually in VPR contribute negatively to the distinctiveness of a place (as shown in Fig. 3). More recently, a CNN able to classify input frames as stable/unstable is trained by Dymczyk et al. [37] for long term visual place recognition. Similar to [36], this work also proposes that vegetation in outdoor scenes is not suitable, but does not consider outdoor dynamic objects like cars, pedestrians, animals etc. Also, informativeness of stable frames in terms of extracted features and predicted matchability is not inspected given that walls are selected as stable elements of an image. Therefore, to the best of author's knowledge, our work combines for the first time all three of these criterion namely memorability, staticity and entropy to create memorable-maps. We show utility of our proposed framework by reporting results on multiple public datasets. The agonstic nature of our framework is presented by using multiple state-of-the-art VPR techniques in combination with memorable-maps.

III. METHODOLOGY
This section presents in depth the framework developed in our work. A sub-section is dedicated to each of the three criterion (i.e. memorability, staticity and entropy) adopted by the framework. Flowchart depicting the working of memorablemaps framework is shown in Fig. 2.
For the purpose of evaluation and analysis, we have used AMOS-Net [13], Hybrid-Net [13] and Region-VLAD [18] as our VPR techniques throughout this paper. The details of these techniques are given in Section IV-B.
A. Memorability 1) Why Memorability?: The human-cognition system is powerful in evaluating what images are useful to be stored in the brain's memory fragments [38] [39]. We usually remember concrete structures like buildings, streets, squares etc. However, more natural scenes like fields, forests, grassy plains and far out sceneries are less memorable. This 'memorability' concept is also intuitive as it is easy to confuse different natural scenes with each other compared to concrete structures. In Fig.  3, we show how such natural scenes are mismatched by stateof-the-art VPR systems leading to false-positives. Each frame first goes through staticity-map computation. Next, we compute the frame's memorability-map and entropy-map which are masked with staticity-map. Finally, we calculate the scores from all three image-maps, threshold and discard confusing frames. 2) Memorability Implementation: Inspired from humanmemorability, we apply the work done originally for marketing and advertising in [7] to VPR problem. A Convolutional Neural Network namely Hybrid-CNN is fine-tuned on LaMem dataset (introduced in [7]). This LaMem dataset is composed of 60,000 images covering multiple scenarios ranging from natural scenery, indoor scenes, outdoor scenes and distinctive objects. The ground-truth human-memorability is computed for each of these images using an interactive game played by multiple human subjects. Images are shown to players in a sequence and are repeated after a random interval where a human has to identify/recall a previously seen image. Resultingly, the output of such fine-tuned Hybrid-CNN is a human-memorablity score m for each input image in the range of 0 − 1; with m = 1 being the most memorable.
The CNN input layer size is set to W 1 × H1 where W 1 = 227, H1 = 227. We re-size every incoming image as follows.
We then split this rescaled-image into 25 non-overlapping crops of 227 × 227 each and sequentially feed them as inputs to CNN. This in-turn gives us the memorability matrix M as shown below.
Where, m i j is the memorability of each 227 × 227 cropped image. To create a memorability-map, we rescale the matrix M from 5×5 to W 2×H2 with bilinear interpolation. An example memorability-map overlayed on an image is shown in Fig.  6. It can be seen (in Fig. 6) that vegetation, natural scenery and trees are identified as less-memorable which is consistent with our motivation in Section III-A1. However, for humancognition (and therefore for the fine-tuned Hybrid-CNN) objects such as faces, animals and vehicles are memorable. But, such dynamic objects are not re-observable and therefore not good for VPR; we cater for this in the following sub-section.

1) Why Staticity?:
The previous sub-section shows how memorability is a good evaluation criterion for a camera frame to be used in VPR. However, one limitation is the fact that objects like cars, pedestrians, buses, animals and bicycles in an image are all classified as highly memorable but are not reobservable (for VPR problem). Resultingly, images that may be memorable but have high dynamic content will fail to match upon repeated traversal. Fig. 4 shows some of these images mismatched by VPR techniques [13] [18].
2) Staticity Implementation: To cater for highly dynamic images, we perform image segmentation into static and dynamic pixels. We re-size all input images to W 2 × H2. We use an object detector [40] that can detect 80 different classes of objects in an image. Out of these 80 classes, 21 correspond to highly-dynamic, commonly-observed objects. These dynamic objects include cars, pedestrians, buses, trucks, animals etc. We, therefore, only consider proposals of bounding boxes coming from objects of interest, i.e., dynamic objects.
The object detector also requires a confidence metric (between zero and one) to give bounding box proposals. We selected this confidence metric as 0.55, which performs well for objects that occupy at least 5% of the total image pixels. Objects occupying less than 5% of the total pixels are ignored by our framework. Hence, by not detecting these low significance objects, the selected confidence metric works in coherence with our criterion. After acquiring bounding boxes which are usually overlapping, we use the algorithm given below to compute staticity-map.  Dynamic   Fig. 6 shows the typical staticity-map computed in our framework. However, although an image containing low-textured scenes (walls/door/pillars) can be classified as concrete (memorable) and static but it does not have distinguishable features, and hence is not distinct. We accommodate this limitation in the following sub-section.
C. Entropy 1) Why Entropy?: An input camera frame containing a room/lift door is commonly observed by a robot navigating indoors. Such a frame is classified as memorable and static, but has little to no information differentiating it from other doors in the building, thus leading to false positives. The same can be extended to any other frame with occlusion resulting from walls, pillars etc. Examples of such confusing frames are shown in Fig. 5. 2) Entropy Implementation: To avoid less informative or occluded frames, we evaluate the information content of an image by computing its local entropy. This local entropy corresponds to the number of bits required to encode the local gray-scale distribution in an image. We use a circular window of 5 pixels radius as our neighbourhood to get the entropy map of an incoming camera frame against each pixel. The total number of histogram bins used for entropy computation are 256 corresponding to 0 − 255 gray-scale intensity values. The generic algorithm for computing this entropy map is shown below and adopted from [41].

Algorithm Computing Entropy Map
Create  (1) and equals 8. Fig. 6 shows an example entropy map computed in our framework.

D. Computing Scores and Thresholding
After acquiring all three maps of an image, we mask memorability map and entropy map with staticity map. This ensures that our decision to select an image based upon memorability and entropy is immune to the information coming from dynamic objects.
Next, we compute the memorability score (MS) of an image as the average value of memorability map and compare it with a memorability threshold (MT ), to evaluate if this image/frame is memorable enough for use in VPR.
Secondly, we compute the percentage of static pixels in our staticity map to get a staticity score (SS). This is then contrasted with staticity threshold (ST ) to decide if an incoming frame has enough static content to be inserted into the map.
Thirdly, we calculate the average value of entropy map and scale it with the maximum value of entropy to get the percentage of information content. This percentage dubbed as the entropy score (ES), is compared with the entropy threshold (ET ) to settle if an input frame has enough information.
Finally, we use a tri-input AND criterion to select images that are memorable, static and information-rich to be inserted into the map.

IV. EXPERIMENTAL SETUP
This section discusses the datasets, VPR techniques and evaluation metric used in our analysis. We present a new dataset ESSEX3IN1, which will be publicly available. Additionally, we briefly discuss three pre-existing public datasets used for reporting our framework's performance. The VPR techniques used for our results and analysis are then summarized. We utlize area-under-the Precision-Recall curve (AUC) which is a well-established performance metric for VPR techniques [42] [17] [14] [12] [18].
A. Evaluation Datasets 1) ESSEX3IN1 Dataset: Most of the Visual Place Recognition datasets have been created from a pre-planned path traversal. Thus, these datasets do not contain confusing images that an exploration robot may come across. Also, these datasets focus on a single type of environment either indoor or outdoor. To evaluate and challenge our framework, we have created a new dataset ESSEX3IN1 which is composed of images from indoor, outdoor and natural scenes.
The dataset was created in two stages at the University of Essex (Colchester Campus) and contains 210 query images and 210 reference images with viewpoint variations. In the first stage, the objective was to take images from all sorts of environments that were either 'confusing' or didn't qualify the human-definition of 'Place'. Two-third of the images in ESSEX3IN1 are from this first stage. The second stage, consists of images that were not confusing and could be defined as places. One-third of the total images are from this second stage. Some images from these stages are shown in Fig. 7.
It is important to note that none of these images were used in tuning our three thresholds and were not seen prior by the proposed framework. The collection of dataset in this twostaged manner was useful for analysis in Section V. 2) Nordland Dataset: The Nordland dataset [21] comprises of a train journey through Norway and is collected in four different seasons with frame-to-frame ground-truth correspondence. We use a subset of this dataset which consists of 1622 query images and 1622 reference images. The query images are from the traversal performed in summer where as the reference images are from winter. Although this dataset does not provide any viewpoint variation, but has significant conditional variation. Some sample images from Nordland dataset are shown in Fig. 8. A retrieved image n is considered true-positive if the original ground-truth is between n − 1 to n + 1 i.e. each query image has 3 ground-truth references.
3) St. Lucia Dataset: The St. Lucia dataset was first introduced in [9]. It was recorded in the surroundings of University of Queensland's St. Lucia campus during multiple times of the day. This dataset consists of moderate viewpoint and illumination variation as shown in Fig. 10. The dataset also contains dynamic objects and scene variation. The groundtruth is derived manually from GPS data such that each query image has three reference images as true-positives.

4) SPEDTest Dataset:
The SPEDTest dataset was introduced in [12] and is a sub-set of the original Specific Places Dataset [13]. It consists of 607 query images coming from a variety of scenes and environments. Frame-to-frame correspondence is available as the ground-truth. Samples images from SPEDTest are shown in Fig. 10.

B. VPR Techniques
We have used three state-of-the-art VPR techniques (namely AMOS-SPP, Hybrid-SPP and Region-VLAD) [13] [18] that have shown promising results in recent research. AMOS-Net is a modified Caffe-Net [43] with all parameters trained on SPED dataset [13]. Hybrid-Net is another modified version of Caffe-Net with weights for top 5 convolutional layers initialized from Caffe-Net [43]. We have used Spatial Pyramidal Pooling as a feature descriptor for both AMOS-Net and Hybrid-Net since it shows excellent results as compared to other feature encoding methods. Features are extracted from 'conv5' layer in case of both Amos-Net and Hybrid-Net. The third VPR technique, Region-VLAD, uses features extracted from selected/interesting regions of an AlexNet pre-trained on Places365 dataset. Vector-of-Locally-Aggregated-Descriptors [44] is subsequently used for encoding the extracted features. In case of Region-VLAD, we use features from 'conv4', number of regions-of-interest as 400 and a visual dictionary size of 128.

C. Evaluation Metric
For evaluating the performance of different VPR techniques AUC has been repeatedly used by the research community. AUC acts as a good metric to assess the performance of a system based on true-positives, false-positives and falsenegatives. However, AUC scores may differ depending on the method employed for computing the area. In our work, we use equation (2) to compute the AUC of a Precision-Recall curve.

V. RESULTS AND ANALYSIS
This section presents the results and analysis in a sequential manner. We first show that images collected from the first stage of ESSEX3IN1 actually lead to poor performance of VPR systems and are not good for insertion into a topological-map. Secondly, we show the segregation performance of proposed framework on these 'confusing' images and 'good' images. Thirdly, we present the AUC improvement of different VPR systems when plugged with our framework on all datasets discussed in sub-section IV-A. This is followed-up with a sub-section dedicated to qualitative analysis showing sample images selected and discarded from all datasets. We then highlight the contribution of each framework criterion and its effect on VPR performance by sweeping within possible range. And finally, we show how such a framework leads to reduced map size and place matching time.

A. Contemporary VPR Systems on ESSEX3IN1 Stage 1
The majority of VPR false positives against ESSEX3IN1 are from the first stage of dataset collection. This is due to the confusing images of fields, trees, doors, cars etc. Some of these false positives are shown in Fig. 11. Fig. 11. VPR false positives upon evaluation on ESSEX3IN1 stage: 1. Images with cars, trees and natural scenes are mismatched. Additionally, images with low information and memorability are almost indistinguishable for even human cognition.
We show the AUC performance of VPR systems separately on Stage 1 and Stage 2 in Fig. 12.

B. Segregation Performance of Proposed Framework
For this sub-section, we apply the proposed memorablemaps framework on complete and randomized ESSEX3IN1 dataset. We use the below thresholds to segregate and discard 'confusing' images from 'good' images.
Memorability-threshold = 0.5 Staticity-threshold = 0.6 Entropy-threshold = 0.4 These thresholds were selected from our analysis on preexisting public VPR datasets. Increasing these thresholds reduces the number of images inserted into the memorablemap. This is shown in Fig. 13 by varying each threshold from 0 − 1 while setting the other two equal to 0. The new database created by presented framework consists of memorable, static and informative images, thus dubbed as a memorable-map. We show [in Fig. 14] how many of the total images selected by presented framework are from which stage of the dataset. By selecting images that are memorable, static and have a higher entropy, the memorable-maps framework gives performance boost to state-of-the-art VPR techniques. Here, we use fixed thresholds, as in previous sub-section V-B, but an AUC sweep across these thresholds is presented later in sub-section V-F. AUC evaluation is performed on the entire (both stages combined randomly) ESSEX3IN1 dataset along with the three public VPR datasets. It is important to note that bad/confusing images found by our framework are not removed from the reference database when evaluating AUC, but are treated as true negatives. This ensures that AUC boost reported here is not due to reduction of reference database size. Fig. 15

D. Selected vs Discarded Images
In this sub-section, we show some images from all four datasets that were selected or discarded by the memorablemaps framework. This gives a qualitative insight into the working of our framework in different environments/datasets. Since the framework evaluates both the query images and reference images, the images in Fig. 19, 20, 21 and 22 are impartial to such distinction.

E. Criterion Contribution Analysis
Each criterion in the memorable-maps framework contributes to AUC boost. This subsection is dedicated to giving an insight into this indiviudal contribution. We use ES-SEX3IN1 for this purpose as it contains confusing images from all three (memorability, staticity and entropy) paradigms. For our AUC evaluation on ESSEX3IN1, we show the contribution of each criterion in Fig. 23. The analysis is performed based on the number of images that were mismatched by a VPR technique and were also discarded by atleast one of the memorable-maps framework criterion.
While Fig. 23 suggests that each of the three criterion namely memorability, staticity and entropy are useful; the % Fig. 19. Images selected and discarded by the memorable-maps framework from complete ESSEX3IN1 dataset are presented. Selected images are predominantly of buildings with distinctive patterns and are largely static. Discarded images consist of far out natural scenes, dynamic objects or have low-entropy. Fig. 20. Images selected and discarded by the memorable-maps framework from Nordland dataset are presented. Selected images consist of either appearing tunnels or bridges which contribute to their distinctiveness. Discarded images consist of vegetation or have low information. Staticity does not play any role in this dataset due to absence of dynamic objects.
contribution is linked to (and can vary with) the number of non-memorable, dynamic and information-less images in the dataset. (See Fig. 13)

F. AUC Sweep Across Framework Thresholds
In this subsection, we present the variation in Visual Place Recognition performance with stricting framework criteria on ESSEX3IN1. We sweep each of the three criterion from 0-1 (Step size: 0.1) while keeping the other two inactive (i.e. set equal to zero). The data points for memorability and entropy thresholds have an upper-bound after which the total number of selected images equals to 0 (refer Fig. 13). Fig. 24 shows that increasing entropy-threshold and memorability-threshold increases the AUC Performance for all three VPR techniques and follows a direct-relationship. On the other hand, the variation in AUC with increasing staticitythreshold follows a different trend. Firstly, the increase in AUC with ST is comparitively lower compared to MT/ET; which is due to the less number of dynamic images in the dataset compared to non-memorable and low-entropy images. Secondly, the variation in AUC with ST for Region-VLAD is higher compared to AMOS-SPP/Hybrid-SPP. We associate this  with the fact that AMOS-SPP/Hybrid-SPP have been trained on SPED (Specific Places Dataset) and discourage features coming from vehicles. While our analysis/results reveal that Region-VLAD extracts and positively matches features coming from cars in different places (See Fig. 4). Thirdly, there is an evident decrease in AUC as ST goes above 0.9. This decrease is expected as images with very low dynamic content can still be matched by contemporary VPR-techniques and discarding such images leads to the observed decline in VPRperformance. The presented trends give a general idea for setting thresholds, thus to maintain a good balance between VPR performance and a salient representation of the world in a metric/topological/topo-metric map.

G. Reduced Map Size and Computational Time
In addition to increase in AUC, the developed framework helps in reducing the robot's map size which has been the motivation for semantic mapping research reviewed in this paper. This size-reduction also leads to reduction in computational overhead for VPR. The reduction in map size for thresholds presented in Section V-B is shown in Fig. 25. The computational performance is reported by calculating the time required to match a query image with all the reference images (having pre-computed feature descriptors) in both a conventional map Fig. 23. Percentage contribution of each criterion into AUC increase is shown for ESSEX3IN1. This contribution is directly linked with the type of environment being explored by a robot. In a highly dynamic environment, the contribution of staticity will be more significant than suggested by this chart and such. and a memorable map. This offline matching time is elaborated in Table 1, where a memorable map having lesser number of reference images (please see Fig. 25) achieves better matching time.

VI. CONCLUSION AND FUTURE WORK
We proposed a cognition-inspired generalized framework for creating 'memorable-maps'. This framework evaluates an incoming camera frame for its memorability, staticity and entropy to decide a frame's insertion into the robot's topological map. By using 'ESSEX3IN1', we show how images that are confusing for human-beings are also mismatched by contemporary VPR systems. The application of proposed framework in detecting these confusing images and subsequently improving VPR performance is presented. We generalize the applicability of our framework by reporting results on multiple public datasets. Due to its agnostic nature, memorable-maps framework can be plugged into any VPR technique giving performance boost.
While presented thresholds are suitable for different indoor, outdoor and natural environments, they are not illumination invariant. In future work, it will be useful to integrate [45] to this work, thus making these thresholds as illuminationdependent variables. This framework coupled with different VPR techniques also enables the creation of a large-scale dataset containing 'good' and 'confusing' images for VPR state-of-the-art. Such a dataset could subsequently help in training an end-to-end neural network for classifying an image as good/bad for map-insertion. We hope that this work draws attention of VPR community towards further research in segregation between confusing and good images. Thus, moving closer to practical deployment of VPR systems. Fig. 24. Variation in VPR AUC performance by changing each of the memorable-maps framework thresholds within their full range on ESSEX3IN1 is presented. Memorability and entropy continously increase AUC until the total number of selected images equals to zero; suggesting that images with higher memorability and entropy are well-matched by VPR methods. On the contrary, since images with low dynamic content should/can still be matched, variation in staticity threshold does not lead to a continous AUC increase. AUC change with ST is not at the same scale as MT/ET so it is shown separately.  25. Reduction in topological map size given similar or better VPR results is of prime importance for an autonomous robot to efficiently map/explore an environment. As depicted here, memorable-maps framework intrinsically reduces map size while giving AUC boost to contemporary VPR systems.