ClipCap: CLIP Prefix for Image Captioning

Image captioning is a fundamental task in vision-language understanding, where the model predicts a textual informative caption to a given input image. In this paper, we present a simple approach to address this task. We use CLIP encoding as a prefix to the caption, by employing a simple mapping network, and then fine-tunes a language model to generate the image captions. The recently proposed CLIP model contains rich semantic features which were trained with textual context, making it best for vision-language perception. Our key idea is that together with a pre-trained language model (GPT2), we obtain a wide understanding of both visual and textual data. Hence, our approach only requires rather quick training to produce a competent captioning model. Without additional annotations or pre-training, it efficiently generates meaningful captions for large-scale and diverse datasets. Surprisingly, our method works well even when only the mapping network is trained, while both CLIP and the language model remain frozen, allowing a lighter architecture with less trainable parameters. Through quantitative evaluation, we demonstrate our model achieves comparable results to state-of-the-art methods on the challenging Conceptual Captions and nocaps datasets, while it is simpler, faster, and lighter. Our code is available in https://github.com/rmokady/CLIP_prefix_caption.

Introduction
In image captioning, the task is to provide a meaningful and valid caption for a given input image in a natural language. This task poses two main challenges. The first is semantic understanding. This aspect ranges from simple tasks such as detecting the main object, to more involved ones, such as understanding the relations between depicted parts of the image. For example, in the top-left image of Fig. 1, the model understands that the object is a gift. The second challenge is the large number of possible ways to describe a single image. In this aspect, the training dataset typically dictates the preferable option for a given image. * Equal contribution.
A politician receives a gift from politician.
A collage of different colored ties on a white background.
Silhouette of a woman practicing yoga on the beach at sunset.
Aerial view of a road in autumn. Many approaches have been proposed for image captioning [4,9,13,19,34,35,42,44,47]. Typically, these works utilize an encoder for visual cues and a textual decoder to produce the final caption. Essentially, this induces the need to bridge the challenging gap between the visual and textual representations. For this reason, such models are resource hungry. They require extensive training time, a large number of trainable parameters, a massive dataset, and in some cases even additional annotations (such as detection results), which limit their practical applicability.
Excessive training time is even more restrictive for applications that require several training procedures. For instance, training multiple captioning models over various datasets could provide different users (or applications) with different captions for the same image. Additionally, given fresh samples, it is desirable to update the model routinely with the new data. Therefore, a lightweight captioning model is preferable. Specifically, a model with faster training times and fewer trainable parameters would be beneficial, especially if it does not require additional supervision.
In this paper, we leverage powerful vision-language pretrained models to simplify the captioning process. More specifically, we use the CLIP (Contrastive Language-Image Pre-Training) encoder, recently introduced by Radford et al. [29]. CLIP is designed to impose a shared representation for both images and text prompts. It is trained over a vast number of images and textual descriptions using a contrastive loss. Hence, its visual and textual representations are well correlated. As we demonstrate, this correlation saves training time and data requirements.
As illustrated in Fig. 2, our method produces a prefix for each caption by applying a mapping network over the CLIP embedding. This prefix is a fixed size embeddings sequence, concatenated to the caption embeddings. These are fed to a language model, which is fine-tuned along with the mapping network training. At inference, the language model generates the caption word after word, starting from the CLIP prefix. This scheme narrows the aforementioned gap between the visual and textual worlds, allowing the employment of a simple mapping network. To achieve even a lighter model, we introduce another variant of our method, where we train only the mapping network, while both CLIP and the language model are kept frozen. By utilizing the expressive transformer architecture, we successfully produce meaningful captions, while imposing substantially less trainable parameters. Our approach is inspired by Li et al. [20], which demonstrates the ability to efficiently adapt a language model for new tasks by concatenating a learned prefix. We use GPT-2 [30] as our language model, which has been demonstrated to generate rich and diverse texts.
As our approach exploits the rich visual-textual representation of CLIP, our model requires significantly lower training time. For instance, we train our model on a single Nvidia GTX1080 GPU for 80 hours over the three million samples of the massive Conceptual Captions dataset. Nevertheless, our model generalizes well to complex scenes, as can be seen in Fig. 1 (e.g., practicing yoga on the beach at sunset). We evaluate our method extensively, demonstrating successful realistic and meaningful captions. Even though our model requires less training time, it still achieves comparable results to state-of-the-art approaches over the challenging Conceptual Captions [33] and nocaps [1] datasets, and marginally lower for the more restricted COCO [7,22] benchmark. In addition, we provide a thorough analysis of the required prefix length and the effect of fine-tuning the language model, including interpretation of our produced prefixes. Overall, our main contributions are as follow: • A lightweight captioning approach that utilizes pretrained frozen models for both visual and textual processing.
• Even when the language model is fine-tuned, our approach is simpler and faster to train, while demonstrating comparable results to state-of-the-art over challenging datasets.

Related Works
Recently, Radford et al. [29] presented a novel approach, known as CLIP, to jointly represent images and text descriptions. CLIP comprises two encoders, one for visual cues and one for text. It was trained over more than 400 million image-text pairs guided by unsupervised contrastive loss, resulting in rich semantic latent space shared by both visual and textual data. Many works have already used CLIP successfully for computer vision tasks that require the understanding of some auxiliary text, such as generating or editing an image based on a natural language condition [5,14,28]. In this paper, we utilize the powerful CLIP model for the task of image captioning. Note that our method does not employ the CLIP's textual encoder, since there is no input text, and the output text is generated by a language model.
Commonly, image captioning [34] models first encode the input pixels as feature vectors, which are then used to produce the final sequence of words. Early works utilize the features extracted from a pre-trained classification network [6,9,13,42], while later works [4,19,47] exploit the more expressive features of an object detection network [31]. Though a pre-trained object detection network is available for the popular COCO benchmark [7,22], it is not necessarily true for other datasets. This implies that most methods would require additional object detection annotations to operate over new and diverse datasets. To further leverage the visual cues, an attention mechanism is usually utilized [4,6,42] to focus on specific visual features. Moreover, recent models apply self-attention [16,43] or use an expressive visual Transformer [12] as an encoder [23]. Our work uses the expressive embedding of CLIP for visual representation. Since CLIP was trained over an extremely large number of images, we can operate on any set of natural images without additional annotations.
To produce the caption itself, a textual decoder is employed. Early works have used LSTM variants [8,38,39], while recent works [16,26] adopted the improved transformer architecture [36]. Built upon the transformer, one of the most notable works is BERT [11], demonstrating the dominance of the newly introduced paradigm. With this paradigm, the language model is first pre-trained over a large data collection to solve an auxiliary task. Then, the model is fine-tuned for a specific task, where additional supervision is used. As our visual information resides in the prefix, we utilize a powerful auto-regressive language model, GPT-2 [30]. Considering the training loss term, earlier works adopt the effective cross-entropy, while contemporary methods also apply self-critical sequence training [15,32,45]. That is, an additional training stage to optimize the CIDEr metric. We deliberately refrain from this optimization to retain a quick training procedure.
Most close to ours, are works that employ vision-andlanguage pre-training to create a shared latent space of both vision and text [19,25,35,46,47]. Zhou et al. [47] use visual tokens extracted from object detector as a prefix to caption tokens. The entire model is then pre-trained to perform prediction utilizing the BERT [11] architecture. Li et al. [19] and Zhang et al. [46] also utilize BERT, but require the additional supervision of object tags. Hence, these methods are limited to datasets in which such object detectors or annotations are available. The approach of Wang et al. [40] mitigate the need for supplementary annotations, but still perform an extensive pre-train process with millions of imagetext pairs, resulting in a lengthy training time. This exhaustive pre-training step is required to compensate for the lack of joint representation of language and vision, which we inherently obtained by employing CLIP.

Method
We start with our problem statement. Given a dataset of paired images and captions {x i , c i } N i=1 , our goal is to learn the generation of a meaningful caption for an unseen input image. We can refer to the captions as a sequence of tokens c i = c i 1 , . . . , c i , where we pad the tokens to a maximal length . Our training objective is then the following: where θ denotes the model's trainable parameters. Our key idea is to use the rich semantic embedding of CLIP, which contains, virtually, the essential visual data, as a condition. Following recent works [47], we consider the condition as a prefix to the caption. Since the required semantic information is encapsulated in the prefix, we can utilize an autoregressive language model that predicts the next token without considering future tokens. Thus, our objective can be described as:

Overview
An illustration of our method is provided in Fig. 2. We use GPT-2 (large) as our language model, and utilize its tokenizer to project the caption to a sequence of embeddings. To extract visual information from an image x i , we use the visual encoder of a pre-trained CLIP [29] model. Next, we employ a light mapping network, denoted F , to map the CLIP embedding to k embedding vectors: Where each vector p i j has the same dimension as a word embedding. We then concatenate the obtained visual embedding to the caption c i embeddings: During training, we feed the language model with the prefix-caption concatenation Our training objective is predicting the caption tokens conditioned on the prefix in an autoregressive fashion. To this purpose, we train the mapping component F using the simple, yet effective, cross-entropy loss: We now turn to discuss two variants of our method regarding the additional fine-tuning of the language model and their implications.

Language model fine-tuning
Our main challenge during training is to translate between the representations of CLIP and the language model. Even though both models develop a rich and diverse representation of text, their latent spaces are independent, as they were not jointly trained. Moreover, each captioning dataset incorporates a different style, which may not be natural for the pre-trained language model. Hence, we propose fine-tuning the language model during the training of the mapping network. This provides additional flexibility for the networks and yields a more expressive outcome.
However, fine-tuning the language model naturally increases the number of trainable parameters substantially. Thus, we present an additional variant of our approach, in which we keep the language model fixed during training. Our attempt to adjust a frozen language model is inspired by the work of Li and Liang [20]. In their work, they accommodate such a pre-trained model to an unfamiliar task by learning only a prefix. Such prefix is automatically optimized to steer the language model towards the new objective during a standard training procedure. Following this approach, we suggest avoiding the fine-tuning to realize an even lighter model, where only the mapping network is trained. As presented in Section 4, our model not only produces realistic and meaningful captions, but also achieves superior results for some of the experiments without finetuning the language model. Note that fine-tuning CLIP does not benefit resulting quality, but does increase training time and complexity. We hence postulate that the CLIP space already encapsulates the required information, and adapting it towards specific styles does not contribute to flexibility.

Mapping Network Architecture
Our key component is the mapping network, which translates the CLIP embedding to the GPT-2 space. When the language model is simultaneously fine-tuned, the mapping is less challenging, as we easily control both networks. Therefore, in this case, we can employ a simple Multi-Layer Perceptron (MLP). We have achieved realistic and meaningful captions even when utilizing only a single hidden layer, as CLIP is pre-trained for a vision-language objective.
Nevertheless, when the language model is frozen, we propose utilizing the more expressive transformer [36] architecture. The transformer enables global attention between input tokens while reducing the number of parameters for long sequences. This allows us to improve our results by increasing prefix size, as shown in Section. 4. We feed the transformer network with two inputs, the visual encoding of CLIP and a learned constant input. The constant has a dual role, first, to retrieve meaningful information from CLIP embedding through the multi-head attention. Second, it learns to adjust the fixed language model to the new data. This is demonstrated in Section. 4, where we offer interpretability for our generated prefix. As can be seen, when the language model is fixed, the transformer mapping network learns a meticulous set of embeddings without any textual meaning. These are optimized to tame the language model.

Inference
During inference, we extract the visual prefix of an input image x using the CLIP encoder and the mapping network F . We start generating the caption conditioned on the visual prefix, and predict the next tokens one by one, guided by the language model output. For each token, the language model outputs probabilities for all vocabulary tokens, which are used to determine the next one by employing a greedy approach or beam search.

Results
Datasets. We use the COCO-captions [7,22], nocaps [1] , and Conceptual Captions [33] datasets. We split the former according to the Karpathy et al. [17] split, where the training set contains 120, 000 images and 5 captions per image. Since COCO is limited to 80 classes, the nocaps dataset is designed to measure generalization to unseen classes and concepts. It contains only validation and test sets, with the training utilizing COCO itself. The nocaps dataset is divided to three parts -in-domain contains images portraying only COCO classes, near-domain contains both COCO and novel classes, and out-of-domain consists of only novel classes. As suggested by Li et al. [19], we evaluate the model using only the validation set. Though some methods utilize object tags of the novel classes, we only consider the setting of no additional supervision, as we find it more applicable in practice. Therefore, we do not employ a constrained beam search [2]. The Conceptual Captions dataset consists of 3M pairs of images and captions, harvested from the web and post-processed. It is considered to be more challenging than COCO due to the larger variety of styles of both the images and the captions, while not limited to specific classes. To focus on the concepts, specific entities in this dataset are replaced with general notions. For example, in Fig. 1, the names are replaced with "politician". For evaluation, we use the validation set, consisting of 12.5K images, as the test set is not publicly available. Consequently, we did not use this set for validation.
Baselines. We compare our method to the state-of-the-art works of Li et al. [19] (known as Oscar), Vision-Language Pre-training model (VLP) [47], and the eminent work of Anderson et al. [4], denoted BUTD. These models first produce visual features using an object detection network [31]. BUTD then utilizes an LSTM to generate the captions, while VLP and Oscar employ a transformer, trained simi-  Photograph of the sign being repaired by brave person.
Globes : the green 3d person carrying in hands globe.
The player staring intently at a computer screen.
The -bedroom stone cottage can sleep people. VLP Actors in a scene from the movie.
The sign at the entrance. Templates: green cartoon character holding the earth globe.
Person works on a video.
The master bedroom has a king -sized bed with a queen size bed. Ours; MLP + GPT2 tuning Actor sits in a hotel room.
The sign at the entrance. 3d render of a man holding a globe.
Person, a student, watches a video on his laptop.
The property is on the market for £ 1.
Ours; Transformer person sitting on a chair in a room.
a sign is seen at the entrance to the store.
stock image of a man holding the earth.
portrait of a young boy playing video game.
one of the bedrooms in the house has been converted into a living room. Figure 4. Uncurated results of the first five images in our test set for Conceptual Captions [33].
A person standing in front of a rock formation in the desert.
A man holding a banana in front of a river.
Two horned goats crossing a road in the desert.
A person sitting at a table with a tray of sushi. larly to BERT [11]. Both VLP and Oscar exploit an extensive pre-trained procedure over millions of image-text pairs. Oscar [19] also uses additional supervision compared to our setting, in the form of object tags for each image.
Our default configuration employs the transformer mapping network, without fine-tuning the language model, denoted Ours; Transformer. Additionally, we also evaluate our variant that utilizes the MLP mapping network, and fine-tunes the language model, denoted Ours; MLP + GPT2 tuning. Other configurations are evaluated in Tab. 1(D).
Evaluation metrics. Similar to Li et al. [19], we validate our results over the COCO dataset using the common metrics BLEU [27], METEOR [10], CIDEr [37] and SPICE [3], and for the nocaps dataset using CIDEr and SPICE. For the Conceptual Captions, we report the ROUGE-L [21], CIDEr, and SPICE, as suggested by the authors [33].
Furthermore, we measure the training time and the number of trainable parameters to validate the applicability of our method. Reducing the training time allows to quickly obtain a new model for new data, create an ensemble of models, and decrease energy consumption. Similar to other works, we report training time in GPU hours, and the GPU model used. The number of trainable parameters is a popular measure to indicate model feasibility.
Quantitative evaluation. Quantitative results for the challenging Conceptual Captions dataset are presented in Tab. 1(A). As can be seen, we surpass the results of VLP, while requiring orders of magnitude less training time. We note that our lightweight model, which does not fine-tune GPT-2, achieves an inferior result for this dataset. We hypothesize that due to the large variety of styles, a more expressive model is required than our light model, which induces a significantly lower parameter count. We compare only to VLP, as the other baselines haven't published results nor trained models for this dataset.
Tab. 1(B) presents results for the nocaps dataset, where we achieve comparable results to the state-of-the-art method Oscar. As can be seen, Oscar achieves a slightly better SPICE score and we attain a slightly better CIDEr score. Still, our method uses a fraction of training time and trainable parameters with no additional object tags required, hence it is much more useful in practice.

GPT-2 tuning
Caption a motorcycle is on display in a showroom.  Figure 6. Prefix Interpretability. We present both the generated caption and our prefix interpretation. Upper: Ours; MLP + GPT2 tuning. Bottom: Ours; Transformer.
Tab. 1(C) present the results for the COCO dataset. Oscar reaches the best results, however, it uses additional input in the form of object tags. Our results are closed to VLP and BUTD which utilize considerably more parameters and training time. Note that the training time of VLP and Oscar does not include the pre-training step. For instance, pretraining of VLP requires training over Conceptual Captions which consumes 1200 GPU hours.
Both Conceptual Captions and nocaps are designed to model a larger variety of visual concepts than COCO. Therefore, we conclude our method is preferable for generalizing to diverse data using a quick training procedure. This originates from utilizing the already rich semantic representations of both CLIP and GPT-2.
Qualitative evaluation. Visual results of the uncurated first examples in our test sets of both Conceptual Captions and COCO datasets are presented in Figs. 3 and 4 respectively. As can be seen, our generated captions are meaningful and depict the image successfully for both datasets. We present additional examples collected from the web in Fig. 1. As can be seen, our Conceptual Captions model generalizes well to arbitrary unseen images as it was trained over a sizable and diverse set of images. We also present in Fig. 5 results over smartphone images, to further demonstrate generalization to new scenarios. Moreover, our model successfully identifies uncommon objects even when trained only over COCO. For example, our method recognizes the wooden spoons or the cake with a candle better than Oscar in Fig. 3, since CLIP is pre-trained over a diverse set of images. However, our method still fails in some cases, such as recognizing the bicycle next to the train in Fig. 3. This is inherited from the CLIP model, which does not perceive the bicycle in the first place. We conclude that our model would benefit from improving CLIP object detection ability, but leave this direction for future work. For Conceptual Captions, our method mostly produces accurate captions, such as perceiving the green 3d person in Fig. 4. As expected, our method still suffers from data bias. For instance, it depicts the bedroom image in Fig. 4 as "The property is on the market for £ 1" after witnessing such captions of property advertising during training.
Language model fine-tuning. As described in Section. 3, fine-tuning the language model results in a much more expressive model, but that is also more susceptible to overfitting, as the amount of trainable parameters increases. As can be seen in Tab. 1, the two variants -with and without the language model fine-tuning -are comparable. Over the extremely complicated Conceptual Captions dataset, we get superior results with the fine-tuning. While over the popular COCO dataset, avoiding the fine-tuning achieves better results. Regarding nocaps dataset, the results are roughly equal, thus the lighter model would be preferable. We thus hypothesize that extremely elaborated datasets or ones that present a unique style require more expressiveness, and hence the more likely it is to benefit from the fine-tuning.
Prefix Interpretability. To further understand our method and results, we suggest interpreting the generated prefixes as a sequence of words. Since the prefix and word embeddings share the same latent space, they can be treated similarly. We define the interpretation of each of the k prefix embeddings as the closest vocabulary token, under cosine similarity. Fig. 6 shows examples of images, the generated captions, and their prefix interpretations. The interpretation is meaningful when both the mapping network and GPT-2 are trained. In this case, the interpretation contains salient words that associate with the content of the image. For Instance, motorcycle and showcase in the first example. However, when we only train the mapping network, the interpretation becomes essentially unreadable since the network is also charged with maneuvering the fixed language model. Indeed, a considerable part of the prefix embeddings is shared across different images for the same model, as it performs the same adjustment to GPT-2.
Prefix length. Li and Liang [20] showed that increasing the size of the prefix length, up to a certain value, improves the performance of the model in an underlying task. Moreover, the saturation length might differ between tasks. For the image captioning task, we conduct an ablation over the prefix lengths using the COCO dataset over two configurations of our method: Ours; Transformer and Ours; MLP + GPT2 tuning. The results are summarized in Fig. 7. For each prefix size and configuration, we train the network for 5 epochs and report the BLEU@4 and CIDEr scores over the test and train sets.
As can be seen in Fig. 7a, increasing the prefix size while allowing tuning of the language model results in overfitting to the training set, due to the large number of trainable parameters. However, when the language model is frozen, we experience improvement for both the training and test evaluations, as can be seen in Fig. 7b. Naturally, extremely small prefix length yields inferior results as the model is not expressive enough. In addition, we point out that the MLP architecture is inherently more limited as it is not scalable for a long prefix. For example, a prefix size of 40 implies a network with over 450M parameters, which is unfeasible for our single GPU setting. The transformer architecture allows increasing the prefix size with only marginal increment to the number of the parameters, but only up to 80 -due to the quadratic memory cost of the attention mechanism.
Mapping network. An ablation study for the mapping network architecture is shown in Tab. 1(C),(D). As can be seen, with language model fine-tuning, the MLP achieves better results. However, the transformer is superior when the language model is frozen. We conclude that when employing the fine-tuning of the language model, the expressive power of the transformer architecture is unnecessary. Implementation details. We used the prefix length of K = 10 for the MLP mapping networks, where the MLP contains a single hidden layer. For the transformer mapping network, we set the CLIP embedding to K = 10 constants tokens and use 8 multi-head self-attention layers with 8 heads each. We train for 10 epochs using a batch size of 40. For optimization, we use AdamW [18] with weight   decay fix as introduced by Loshchilov et al. [24], with a learning rate of 2e −5 and 5000 warm-up steps. For GPT-2 we employ the implementation of Wolf et al. [41].

Conclusion
Overall, our CLIP-based image-captioning method is simple to use, doesn't require any additional annotations, and is faster to train. Even though we propose a simpler model, it demonstrates more merit as the dataset becomes richer and more diverse. We consider our approach as part of a new image captioning paradigm, concentrating on leveraging existing models, while only training a minimal mapping network. This approach essentially learns to adapt existing semantic understanding of the pre-trained models to the style of the target dataset, instead of learning new semantic entities. We believe the utilization of these powerful pre-trained models would gain traction in the near future. Therefore, the understanding of how to harness these components is of great interest. For future work, we plan to incorporate pre-trained models (e.g., CLIP), to other challenging tasks, such as visual question answering or image to 3D translation, through the utilization of mapping networks.


An Effective Hotel Recommendation System through Processing Heterogeneous Data †

: Recommendation systems have recently gained a lot of popularity in various industries such as entertainment and tourism. They can act as ﬁlters of information by providing relevant suggestions to the users through processing heterogeneous data from different networks. Many travelers and tourists routinely rely on textual reviews, numerical ratings, and points of interest to select hotels in cities worldwide. To attract more customers, online hotel booking systems typically rank their hotels based on the recommendations from their customers. In this paper, we present a framework that can rank hotels by analyzing hotels’ customer reviews and nearby amenities. In addition, a framework is presented that combines the scores generated from user reviews and surrounding facilities. We perform experiments using datasets from online hotel booking platforms such as TripAdvisor and Booking to evaluate the effectiveness and applicability of the proposed framework. We ﬁrst store the keywords extracted from reviews and assign weights to each considered unigram and bigram keywords and, then, we give a numerical score to each considered keyword. Finally, our proposed system aggregates the scores generated from the reviews and surrounding environments from different categories of the facilities. Experimental results conﬁrm the effectiveness of the proposed recommendation framework.

Introduction
Recommendation systems play a vital role in making suggestions for items. They are used to filter information from different networks and predict the output based on the user's preferences. These systems have become extremely popular, and a relevant application of recommender systems is the travel industry. A large number of travel industries are benefiting from the recommendation systems in improving customer satisfaction and experience. In this way, they are making massive chunks of revenue, which is why most of them are turning to recommendation systems. In this paper, one of the main goals of our proposed approach is to provide a platform considering the analysis of the reviews of the customers and the surrounding facilities of the nearby areas of the hotels. Extraction of features from reviews is necessary for providing better recommendations.
Hotel reputation these days is strongly affected by the ratings provided by the guest [1]. Actually, guests are highly appreciated to rate hotels and comment on different aspects of the hotels. Online reviews provided by the customers have a significant impact on hotel revenues [2]. Customers' trust has become a crucial factor when making decisions for online hotel booking. There has been an increasing effort in the current state-of-the-art literature [1][2][3][4][5][6][7] to analyze hotel reviews and ratings in the last decade. In this paper, we build a framework to generate scores from hotel reviews and ratings. We also consider the impact of nearby amenities of the hotels. Hotel selection heavily depends on the different types of P.O.I. (Points of interest), such as public transport, food, and shops. Figure 1 shows a comparative analysis of the overall ratings of a specific hotel for three different hotel booking websites. Ratings vary from website to website. One hotel which is considered average in terms of ratings in one of the hotel booking websites can be found better in other hotel booking websites. For this reason, understanding which point of interests are influencing the hotel reviews is difficult from the descriptions of the text. So, the recommendation generations by analyzing texts are not sufficient enough. In our proposed system, we considered the nearby P.O.I.s of the hotels by using Google Place API. Our system can rank hotels in four different ways considering (1) reviews and comments, (2) surrounding environments of the hotels, (3) numerical ratings, and (4) our proposed aggregated scores. Heterogeneous data are an unstructured data type which means a massive amount of data in diverse formats or nature. These unstructured data include text, numbers, images, demographic data, etc. Hotel booking websites contain this type of data. The analysis of the scores generated from the hotel reviews and surrounding P.O.I.s is necessary. We consider data from two famous hotel booking websites. The experimental outcomes give valuable insights into the viewpoints of the guests of the hotels. Figure 2 shows the surrounding facilities for a specific hotel for two widely used hotel booking platforms.
A comparative analysis of some reviewers' comments for two different hotel booking websites, i.e., TripAdvisor and Booking are shown in Table 1. The textual reviews can provide opinions, contextual information for recommender systems. For example, based on the reviews of the customers who stayed at the hotels, a recommender system can recommend a hotel which the previous customers liked.
The key contributions of this paper are as follows: • We propose a hotel recommendation framework which is implemented by analyzing the (1) reviews generated by the customers of the hotels, and (2) nearby amenities of the hotels; • The proposed framework computes scores from the customers' reviews and the nearby amenities of the hotels; • The proposed method can be helpful for decision-makers, managers of the hotel industry to consider P.O.I.s, review scores for ameliorating the hotel recommendation except for the specific rating score; • We consider data from multiple sources such as Tripadvisor and Booking.  Table 1. A comparative analysis of some reviews of two specific hotels for two different hotel booking websites , i.e., Tri-pAdvisor [8] and Booking [11]. (2) Great staff -lovely atmosphere -delicious breakfast -perfect location. Would recommend. Our bathroom was a little dated -the shower in particular was not good.

HID
The paper is organized as follows. Section 2 overviews the related literature review. In Section 3, we present the architecture of the proposed hotel recommendation system. The experimental outcomes are presented and discussed in Section 4. Finally, Section 5 concludes our work and discusses the future research directions.

Related Work
Data over the internet is growing so fast as the people's option to express their views about products or services is increasing rapidly. Due to the growing diversity of data generated from hotels worldwide, a turn of attention has been observed in recent studies in adopting numerous ways of managing these valuable data. In [3], they used a big data solution involving Hadoop to deal with the variety of numeric data as well as textual data in the heterogeneous form. Sharma et al. [12] used NLP (Natural Language Processing) in their work to determine the rating of the hotel used by the previous customers. The authors of [13] proposed the use of a unified deep NLP model, which analyzes sentences in reviews. They make use of BERT embedding to transform the raw text data into a unified review-POI latent space. It is necessary to extrapolate useful and essential information due to the potential effect that customer's opinions can have on businesses. Most of these data are textual data accompanied by a specific numerical rating. To increase customer satisfaction, researchers are building systems that can extract and leverage the knowledge from such reviews to offer guidance on the selection of hotels. The reviews typically contain the customers' opinions on the hotels and ratings, which indicate the sentiment towards the accommodation and fully characterizes the experience itself.
In [14], an approach to recommend hotels to the users by considering nearby facilities of the hotel was presented. The approach utilized the P.O.I. (Points of Interest) database to obtain the nearby amenities of the hotels. It measured the accommodation preferences of the users by using the reviews provided by the users and calculated the similarity score between the hotels and user preferences by using a similarity measure technique. The top-k hotels are suggested and recommended to the user. The experiments used a dataset collected from TripAdvisor. In [13], the authors proposed the use of a unified deep natural language processing (NLP) model which analyzes sentences in reviews and uses public TripAdvisor hotel-review datasets to validate the approach experimentally. They addressed the challenge of investigating the similarities and dissimilarities between cities by considering the textual reviews and numerical ratings of the hotels and their correlation with the nearby P.O.I.s. They performed their experiment on public TripAdvisor hotel-review datasets and the results provided valuable insights into the viewpoints of hotel guests and suggested further investigation in this direction. Yang et al. [15] presented their effort at constructing a location-aware recommendation system that can model user preferences mainly based on the reviews of the users. They used datasets provided by Yelp. However, they have only included the textual reviews to grasp the nature of people's preference. Yang et al. [16] classified three different categories by considering all location-related factors. The three categories are accessibility to P.O.I., transport convenience, and the surrounding environment. The results confirmed that the presence of airports, public transport, attractions, universities, etc., are significant determinants. Chen et al. [17] combined the conventional recommendation technology with location-based services to provide recommendations. They considered price, service, the location of the tourist, etc., to provide recommendations. The results provided by their system can be nearest to the tourists' needs. The use of location-based social services has enabled opportunities for providing better services through P.O.I. P.O.I. recommendation is personalized, location-aware, and context-depended, unlike traditional recommendation tasks. Recently, many attempts have been dedicated to capturing user preference data from textual reviews for rating prediction purposes [18]. The critical challenge is to understand the key factors that contribute to customer dissatisfaction or satisfaction employing datadriven approaches [19]. Brett et al. [2] showed that the positive rating on customer actions is more influential than advertising strategies. So, review analysis and extraction of the hidden knowledge from reviews is particularly appealing.
Ramzan et al. [3] proposed a recommendation system that helps users find hotels by considering heterogeneous data. The experiments used two different hotel booking datasets that contain reviews, ratings, and ranks to represent data heterogeneity. Their proposed system generates polarity scores from the reviews by using NLP techniques and calculates the aggregated polarity score for each feature based on the reviews from selected websites. By aggregating numerical scores provided by ratings and polarity scores, it generates recommendations to the users. Final recommendations are generated by applying the fuzzy logic approach. Both qualitative and quantitative features of likeness can be achieved by using not only ratings but also reviews of the texts.
In [20], a text to score generation algorithm is proposed, which considers some keywords and their corresponding scores to generate scores from the reviews. They only used unigram keywords, and thus, pairwise combinations of words are neglected. Compared with their work, we consider the combination of unigram and bigram keywords. In our system, both single words and a pairwise combination of words are considered. Sharma et al. [12] examined a recommendation system by using a multi-criteria reviewbased approach. The approach is based on the user's reviews and preferences. They use various NLP approaches to find out the rating of a hotel from previous users. Instead of simple star ratings, their system also deals with the process to suggest hotels based on multi-criteria ratings. These ratings are derived from textual reviews. They only consider the TripAdvisor dataset for their experimental purpose.
In [21], a new feature and opinion extraction method based on the characteristics of online reviews was proposed to extract the user opinions from the user reviews effectively. They crawled a real online restaurant review dataset and collected 54,208 reviews. They selected 4000 reviews randomly and features and opinions extraction from these reviews are done manually. However, these systems process only homogeneous data, whereas most of the data on the web are heterogeneous. Chuhan et al. [6] have tried to express user preferences comprehensively by jointly analyzing hotel ratings and customer reviews. Zhang and Mao [22] suggested that appropriate recommender systems should be developed to achieve true and relevant recommendations according to the choice and preferences of the customers. The deviation of various approaches, objective and advantages of the various recommendation systems are shown in Tables 2 and 3, respectively.

References
Objective Advantages [14] Recommend hotels by using surrounding environments of locations To provide recommendations by considering the nearby amenities of the hotels [21] To propose a new feature and opinion extraction method from online reviews Their proposed method combine user preference and opinion for recommendation [3] To provide true recommendations considering multiple types of data Recommend hotels by using heterogeneous data (Ranks, ratings and reviews) [15] Present a user preference based RS A P.O.I. based recommendation system that models the preferences of the users' by considering user reviews [12] To select the best suited hotel in a city according to user reviews and preferences To generate hotel recommendations based on multi criteria settings

Proposed System Architecture of Hotel Recommendation System
In this section, we will elaborate on the architecture of our hotel recommendation system. Our system contains the following modules: data pre-processing, storage, surrounding environment's evaluation, review analysis, and recommendation generation. Figure 3 shows our system architecture. In the review analysis module, scores are generated from pre-processed textual reviews. Score generation procedures from the nearby amenities of the hotels are performed in the surrounding environment's valuation module.

Dataset Description
We used two different hotel booking datasets for our experimental purposes. The datasets we used in our work are publicly available. We used the framework where the pre-processing stage is performed to the raw sentences, making it more understandable. The first dataset we used in our experiment was collected from Kaggle. This dataset contains about 515 K customer reviews and scoring of 1493 luxury hotels across Europe. For further analysis, geographical locations of hotels are also included here [23]. Table 4 shows the description of the dataset attributes. The file contained 17 attributes.
Another dataset is used for the reviews of hotels collected from TripAdvisor (259,000 reviews). This dataset was initially used for opinion-based entity ranking. We collected this dataset from [24]. We considered 875 hotels of London from these large datasets. We created a CSV file where we manually assigned a unique hotel ID for each hotel for our experimental purpose. The CSV file contains five fields which are shown in Table 5. By using Google API, we collected all considered facilities in the nearby area of the hotels. Our system categorizes the nearby amenities of each hotel by using different categories of the category tree shown in Figure 4.

Hotel_Address
Address of the hotel Avg_Score The average numerical scores of the corresponding hotels HName Name of the corresponding hotel Negative_Review Review (Negative) given by the reviewers Positive_Review Review (Positive) given by the reviewers Total_Number_of_Reviews Total number of valid reviews is represented by it lat It represents the latitude information of the hotel lng It represents the longitude information of the hotel Table 5. Description of OpinkRank Hotel Dataset.

Attributes Description
HID It represents a unique Id for each hotel Hotel_Name It represents the hotel name Review_Title It represents the title of a review Full_Review It represents the full reviews given by each individual users

Storage Module
The storage module preserves the necessary information's for the purpose of generating recommendations. The four storages we used in our system and their functions are given below: • User review database is used to store the textual reviews of the customers; • Keyword database is used to store the extracted keywords for the purpose of score generation; • P.O.I. database is used to store geolocation data about the nearby amenities of the hotels; • A numerical rating database is used to contain the numerical ratings from hotel booking websites.
While some information may be put to use immediately, much of it will serve a purpose later on. When data are properly stored, the data can be quickly and easily accessed in the time of need. We use SQL (Structured Query Language) to store the data.

Data Pre-Processing
Data pre-processing is the process of removing incomplete and noisy data to clean data and put them in a formatted way while doing any operation with them. The kind of data we used in our work contains symbols and unusual text that need to be cleaned. Datasets may be of different formats for different purposes. We usually put the data into a CSV file.
Algorithm 1 shows our data pre-processing algorithm. Our algorithm is implemented in Python which is a high-level programming language and has a great number of dataoriented feature packages. These packages can speed up and simplify data processing, thus making it time-saving. In addition, it also has many excellent libraries for data analysis. Python can handle large datasets; it can more easily implement automated analysis. The pre-processing includes the steps of data integration, removal of missing values, removal of stop words, conversion to Lowercase, Tokenization, removal of special characters and digits, parts of speech tagging, lemmatization, etc.

Review Analysis Module
The textual data need to be processed in order to retrieve more specific opinions. The keywords we consider in our system are categorized into ten different categories. The scores are calculated from the reviews of the customers. Table 6 lists some examples of keywords of different categories. The review-to-score generation procedure is shown in Algorithm 2.
The scores are calculated for a single review of a hotel by using the following Equations (1) and (2): For each unigram/bigram keyword found in the review, multiply the keywords score (weight(k i )) with the number of occurrences of the keyword present in the review. Then, total scores are generated by aggregating the scores considering the effect of n number of unigram/bigram keywords present in the review. The review score is computed by the following Equation (3): The total score generated by considering all of the k reviews of a particular hotel is computed by using Equation (4) given below: The total review score is computed by aggregating all k review scores. The average review score generated for a single hotel is computed by using Equation (5) given below: Scores Generated f or a Single Hotel = Total Review Score k An average score is calculated for a single hotel by dividing the total review scores generated from all k reviews to the value of k. Total Score = Scoreuni + Scorebi; end Overall_score = Total Score / count scores_for_review=scores_for_review + Overall_score no_of_reviews + =1 end Average_score=scores_for_review/no_of_reviews

Evaluation of Surrounding Environments
The P.O.I.s (Points of Interest) database is used in our system to evaluate the surrounding environments of the hotels. Using Google Place API, our system collected all considered facilities within five hundred meters of each hotel. We choose five hundred meters for our experimental purpose. By using a Category Tree (CT) shown in Figure 4, we classified different facilities into eight different categories. The internal nodes represent the types of facilities. The leaf nodes denote the objects of the facilities. Our system generates scores from the surrounding contexts of the hotels based on the information of the CT. The procedure of the surrounding environment's evaluation is shown in Figure 5. Our considered eight categories are shown in Table 7. Total scores are generated by aggregating the scores generated by all of the categories. Now, assume that there are two airports, four restaurants, one university, one movie theater, one bus station, and one night market within five hundred meters from a specific hotel. Looking at the CT of Figure 4, we can see that two airports and one bus station are within the category "Travel and Transport", four restaurants are inside the category "Food", one university inside the category "College and University", one movie theater within the category "Arts and Entertainment" and one night market within the category "Nightlife Spot". In Figure 6, for different categories of surrounding facilities, the number of facilities is shown for a specific hotel H 1 . Here, the number of facilities of H 1 for C 1 is 1, C 2 is 1, C 3 is 3, C 4 is 0, C 5 is 4, C 6 is 1, C 7 is 0, and C 8 is 0.   The scores are calculated for a single category are measured by using Equation (6): Here, n denotes the total number of sub-categories for a specific category. F jk represents the total number of facilities of type k for sub-category j. In our proposed method, we consider two types of weights, so the value of l is 2.
W k represents the weight of the facility type; C i represents the ith category; and S.E.S. represents the surrounding environments score.
The scores are calculated for all of the categories are measured by using Equation (7): We give +1 score for the most important facilities and 0.5 for the other facilities. After determining the surrounding facilities of a hotel, the scores are generated by using Equations (6) and (7). Each of the considered categories are divided into some or many sub-categories. The overall surrounding environment score of a hotel is determined by aggregating the scores generated from all of the categories for that hotel. The scores are generated for each of the considered sub-category. Let us assume that there is a hotel which has 10 facilities in its surrounding areas within a specific range. Among them five facilities are under the category "Arts and Entertainment" and another five are in the category "Food". Then, the scores are calculated by adding the results obtained from the surrounding environment scores of all considered categories. There can be two or more sub-categories for each of the categories. For each sub-category, there are two types of weights we consider for the facility. The most important facilities are considered as type-1 facility and other facilities are considered as type-2 facility. For a specific category, scores are generated by adding the surrounding environment scores of all of the sub-categories of the considered category. The surrounding environment score of a specific hotel is calculated by using Equation (7).

Recommendation Generation Module
The recommendation generation module generates recommendation by aggregating the scores generated from reviews and nearby amenities of the hotels. The aggregated score is the summation of S.G.R. (Score Generated from Review) and S.E.S. (Surrounding Environments Score). The scores are calculated by our system for a specific hotel that contains n number of reviews is computed by using Equation (8) given below:

Experimental Results and Analysis
The top-10 recommendations based on different settings and using average numerical ratings of hotel bookings are discussed here. The dataset we considered here is collected from [23]. This dataset contains information on 1493 hotels. From Table 8, we can see that "Ritz Paris" is the topmost hotel by using average numerical ratings of Booking. The numerical rating score obtained for this hotel is 9.8. We can also see the top-10 recommended hotels by analyzing the reviews of the reviewers in Table 9. Table 8. Top-10 recommended hotels for the 1493 hotels of booking.com based on the average numerical ratings of booking.com.

S. No.
Ratings Hotel Name By considering nearby amenities of the hotels, the top-10 recommended hotels for the 1493 hotels of booking are shown in Table 10. Finally, the top 10 recommendation generation based on our system is shown in Table 11. By using our developed RSG algorithm, our system generates scores from the reviews. The highest score obtained from the average review scores of each hotel is 6.91. The name of the hotel is "South Place Hotel". Next, our system analyzed the nearby amenities of the hotels. From Table 10, we can see that "Hotel Kaiserin Elisabeth" is the highest-ranked hotel. Finally, our system computes the aggregated scores of each considered hotel.
From Table 11, we can see that "Hotel Kaiserin Elisabeth" has the highest ranked hotel and the score generated for this hotel is 28.11. The "Hotel Casa Camper" is ranked as fourth by ratings of Booking but it is ranked as ninth by analyzing reviews. From Tables 12-14, the top-10 recommendation generation based on the different settings are shown. Top-10 recommendation generation uses the following parameters: review scores generated by using our developed RSG algorithm, scores generated from nearby amenities of the hotels and scores generated by our system. The TripAdvisor dataset we considered here is collected from [24]. Table 9. Top-10 recommended hotels for the 1493 hotels of booking.com by analyzing reviews.

S. No.
Scores Generated from Reviews Hotel Name When selecting a hotel for staying purposes, hotel attractions are very important as most customers of the hotels are tourists. Hotel review analysis is also very essential for the customers as well as the surrounding environments of the hotel. If two hotels have the same ratings, then from review scores, surrounding environments scores, a better decision can be taken by the customers. The rankings of the hotels by the surrounding environments can be important for someone who is only interested in the surrounding facilities of the hotels. Someone who is influenced by only the reviews of the previous customers, then, the review scores can be important to him/her. Scores generated from reviews reflect the opinions of the customers of the hotels and the scores generated from surrounding environments reflect the surrounding facilities of the nearby areas of the hotels. The integrated scores generated by our system are a different way of providing recommendations to the customers. The integrated score is the reflection of both review and surrounding environment scores. Table 11. Top-10 recommended hotels for the 1493 hotels of Booking by using our system.

S. No.
Scores Generated by Our System Hotel Name There are 214 hotels that are common in the dataset of both of the hotel booking websites. Top-10 recommendation generation based on average numerical ratings of Booking is shown in Figure 7. Considering the two datasets of the common hotels, the top-10 hotels recommended by our system are shown in Figures 8 and 9, respectively. From Figure 7, we can see that "Haymarket Hotel" is ranked as 2nd by average numerical ratings of Booking. It is ranked as 3rd by considering the dataset of TripAdvisor and it is ranked as 5th by considering the dataset of Booking. From Figures 8 and 9, we can also see that "Hilton London Tower Bridge", "London Marriott Hotel County Hall", and "Cavendish Hotel" are also included in the top-10 recommended hotels by considering the dataset of both hotel booking websites.   There are 214 hotels which are common in both of the hotel booking datasets. The recommendation time of both of the hotel booking datasets for the selected 214 common hotels is given below: We have compared the execution time of our proposed method with that of Liu et al. [21]. The execution time of our proposed method for the 214 common hotels by considering the data of both hotel booking websites is shown in Table 15. The runtime comparison of our proposed method with [21] is shown in Figure 10. The total execution time found in the method of [21] was about 27 s, whereas that of our method was about 6.55 s and 12.46 s for the considered two datasets, respectively. The reason for this difference is that they proposed a method for opinion-feature extraction from online reviews. They randomly selected 4000 reviews and manually extracted features and opinions from these reviews. The execution time of our method is less than that reported in [21]. The reason is that our system generates scores by considering the impacts of different important keywords present in the review and uses the RSG algorithm. As opinions may vary a lot in the reviews from different domains, the extraction is challenging and time-consuming. Experimental results show the effectiveness of the proposed recommendation method.

Discussions
In this paper, we proposed a hotel recommendation system that considers the reviews of the reviewers collected from two famous hotel booking websites. Our proposed framework consists of a data storage module, review analysis module, surrounding environments evaluation module, data processing module, and recommendation generation module. To generate scores from the reviews of the hotels, we developed an RSG algorithm, which takes input as review text and generates scores by considering the impact of both single keywords and a pairwise combination of keywords as outputs. Then a method is used to generate scores by considering the nearby amenities of the hotels. By using Google Place API, the nearby amenities of the hotels are collected. The nearby amenities of hotels are categorized into eight different categories. The scores generated for each of the categories of hotels are aggregated. Then, by using our developed RSG algorithm, scores are generated from the reviews. Some hotel booking systems are available in the state-of-the-art for providing recommendations to the users. Our proposed framework considers the hotels' nearby amenities and analyzes reviews to generate better user recommendations. The data we used in our work were collected from two famous hotel booking websites, i.e., TripAdvisor and Booking, respectively.

Conclusions and Future Research Directions
With the increase of applications using the Internet, the sources of data are getting richer in heterogeneity. Therefore, the various factors in the new data bring new challenges. However, it is also a chance to create novel methods to achieve better recommendation results. So, for this reason, in this paper, we consider heterogeneous data to generate hotel recommendations for the users.
We proposed a hotel recommendation framework to predict top-rated hotels based on the scores generated from reviews and nearby amenities of the hotels through experimental analysis. We have used two reliable data repositories, TripAdvisor and Booking, containing a significant number of numerical ratings, textual reviews, geolocation information, to represent the heterogeneity of data. After data pre-processing, our system generates scores from the reviews of the selected hotel booking datasets. Review scores are aggregated with the surrounding environment scores of the hotels. These heterogeneous data sources, such as ratings, textual reviews, and P.O.I.s are used in our proposed approach, and final aggregated scores are obtained as shown in the experimental results section. The rank of the topmost hotels by using the final aggregated scores are shown for different datasets in the experimental results section. We compared the results of our proposed system with the top-10 results produced by the baseline hotel booking website. In most of the existing recommendation systems, hotel ratings and rankings are typically calculated based on the reviews of previous users only, without considering the hotels surrounding environments.
When selecting a hotel for staying purpose, hotel attractions, such as tourist areas, shopping services, nightlife spots, restaurants, transportation, etc., are very important. More specifically, as most customers of the hotels are tourists, there is a need to consider the location of the hotels. Hotel review analysis is also very essential for the customers as well as the nearby amenities of the hotel. Hotel reviews shed light on the behaviors that had been perceived as pleasing or unpleasing by hotel customers. The proposed system can be helpful to the decision-makers, managers, etc., of the hotel industry to analyze online reviews on a regular basis for ensuring users' satisfaction. The proposed recommender system suggests the decision-makers of the hotels to consider the reviews, P.O.I.s, ratings, and the integration of P.O.I.s, review scores to improve the hotel recommendation systems. Our system can also help customers select the best-matched hotels when there are several hotels of the same category based on some features such as rank.
In the future, we will study methods and techniques which will improve our recommendation systems, and we will try to design the recommender system in a way that will consider dynamically updated data containing the reviews to provide better recommendations to the users. So, for example, the hotels which have improved their facilities after receiving low reviews will be considered. Another direction for future research might be using more data from different sources with different formats. Although a large-scale dataset was used in this paper for generating recommendations, more data with different parameters from other sources can be definitely helpful.


extfieldsofstudy:
Medicine
Computer Science

text:
A Biomedical Knowledge Graph System to Propose Mechanistic Hypotheses for Real-World Environmental Health Observations: Cohort Study and Informatics Application

Background Knowledge graphs are a common form of knowledge representation in biomedicine and many other fields. We developed an open biomedical knowledge graph–based system termed Reasoning Over Biomedical Objects linked in Knowledge Oriented Pathways (ROBOKOP). ROBOKOP consists of both a front-end user interface and a back-end knowledge graph. The ROBOKOP user interface allows users to posit questions and explore answer subgraphs. Users can also posit questions through direct Cypher query of the underlying knowledge graph, which currently contains roughly 6 million nodes or biomedical entities and 140 million edges or predicates describing the relationship between nodes, drawn from over 30 curated data sources. Objective We aimed to apply ROBOKOP to survey data on workplace exposures and immune-mediated diseases from the Environmental Polymorphisms Registry (EPR) within the National Institute of Environmental Health Sciences. Methods We analyzed EPR survey data and identified 45 associations between workplace chemical exposures and immune-mediated diseases, as self-reported by study participants (n= 4574), with 20 associations significant at P<.05 after false discovery rate correction. We then used ROBOKOP to (1) validate the associations by determining whether plausible connections exist within the ROBOKOP knowledge graph and (2) propose biological mechanisms that might explain them and serve as hypotheses for subsequent testing. We highlight the following three exemplar associations: carbon monoxide-multiple sclerosis, ammonia-asthma, and isopropanol-allergic disease. Results ROBOKOP successfully returned answer sets for three queries that were posed in the context of the driving examples. The answer sets included potential intermediary genes, as well as supporting evidence that might explain the observed associations. Conclusions We demonstrate real-world application of ROBOKOP to generate mechanistic hypotheses for associations between workplace chemical exposures and immune-mediated diseases. We expect that ROBOKOP will find broad application across many biomedical fields and other scientific disciplines due to its generalizability, speed to discovery and generation of mechanistic hypotheses, and open nature.

Introduction
"Knowledge graphs" (KGs) have become a common approach for knowledge representation across scientific disciplines, including biomedicine [1]. In a KG, multiple expert-curated "knowledge sources" are integrated into a graph structure, with nodes representing entities and edges providing the relationship between nodes. Within a biomedical KG, the nodes represent biomedical entities, such as drugs or diseases, and the edges describe relationships that connect the nodes, such as treats (eg, drug treats disease). The curated knowledge sources that populate a biomedical KG include both databases, such as DrugBank [2] and Comparative Toxicogenomics Database (CTD) [3], and ontologies, such as Monarch Disease Ontology [4] and Human Phenotype Ontology [5]. Various reasoning tools and inferential algorithms are typically applied to KGs [1], thus allowing users to construct complex queries that ask, for example, if gene X is connected to both chemical exposure Y and disease Z, then the protein product of gene X may represent a potential drug target. Indeed, successful applications of KGs include drug repurposing [6] and the identification of new drug targets [7].
While KGs, such as Monarch, are openly available, many of the more sophisticated KGs remain proprietary. Perhaps the most well-known proprietary KG is the Freebase-derived Google KG that powers Google's web search capability [8]. As part of the Biomedical Data Translator program [9][10][11], we have developed an open KG-based biomedical system termed Reasoning Over Biomedical Objects linked in Knowledge Oriented Pathways (ROBOKOP) [12,13]. ROBOKOP is designed to answer questions such as what genes are associated with recovery from COVID-19 infection? why is imatinib effective in the treatment of asthma? what biological pathways are associated with stroke-related morbidity? Note that these questions imply complex mechanistic relationships between terms in the query such as drug and disease. The ROBOKOP KG is designed to provide answers in the form of putative mechanistic pathways connecting the query terms.
Herein, we provide an overview of ROBOKOP and its real-world application to data derived from the Environmental Polymorphisms Registry (EPR) within the National Institute of Environmental Health Sciences. Specifically, we focused on an EPR study that aimed to explore the impact of workplace exposures on immune-mediated diseases (IMDs) such as asthma, allergy, multiple sclerosis, rheumatoid arthritis, and ulcerative colitis. We first conducted an exploratory analysis of self-reported exposures and IMD symptoms in order to identify significant associations between workplace chemical exposures and IMD. We then used ROBOKOP to (1) validate statistically significant associations by determining whether plausible connections exist within the ROBOKOP KG and (2) propose biological mechanisms that might explain the associations and serve as hypotheses for subsequent testing.

ROBOKOP
ROBOKOP is a biomedical KG-based question-answering system that is comprised of both a front-end user interface (UI) and a back-end KG, both of which are openly available [12][13][14][15]. The ROBOKOP KG uses the Biolink model [16] as an upper-level ontology that can be applied to express domain knowledge as a graph of relationships between biomedical entities. The ROBOKOP KG currently contains 6 million nodes and 140 million edges, with nodes representing a wide range of biological entities, such as genes, biological processes, anatomical features, diseases, and phenotypes, and edges representing predicates, such as is associated with, causes, and increases expression of. The ROBOKOP KG is derived from over 30 curated biomedical data sources (Multimedia Appendix 1) that have been integrated into a graph structure ( Figure 1). The curated data sources were openly available and accessible via direct import into a local Neo4j instance. Some of the data sources were only partially complete and/or required preprocessing. showing classes of nodes as defined by the Biolink model. Note that the schema provides a user guide to assist with the proper construction of queries by providing a visual overview of nodes that are connected in the ROBOKOP knowledge graph.
The ROBOKOP UI allows users to posit queries and quickly explore ranked-and-scored answer sets or subgraphs. ROBOKOP queries are meta-graphs [17][18][19] or question graphs of the entity types shown in Figure 1. The meta-graph or machine question has a general structure that is defined by the user on the basis of the high-level question of interest. Users can select nodes as named entities specified as a CURIE (Compact Uniform Resource Identifier) or specified only as the entity type. A drop-down menu provides users with choices to free-text node entries. Answers are structured as subgraphs that match the query in topology and type, as well as any desired properties of nodes and edges.
A given query will frequently produce many answers or subgraphs, especially for queries with little specification regarding nodes and edges or with multiple nodes and edges. As such, the ranking of subgraphs by relevance to the query and the strength of the supporting evidence are critical for user exploration of results. The ROBOKOP answer-ranking algorithm [13] weighs each edge within each subgraph using a metric that is based on the number of PubMed abstracts that cite both the source and target nodes. Publication support and predicate assertions are provided by the curated knowledge sources used to create the KG (Multimedia Appendix 1). Additional publication support is provided by a ROBOKOP service termed OmniCorp [20], which contains a graph of PubMed identifiers linked to concepts (ie, potential nodes in the ROBOKOP KG) referenced within abstracts. OmniCorp is built by processing all PubMed abstracts with the SciGraph Named Entity Recognition application programming interface [21] and matching text in titles and abstracts to concepts from a predetermined set of biological ontologies. The ROBOKOP answer-ranking algorithm calculates a confidence score for each answer subgraph with respect to the distance between leaves of the answer subgraph, considering the edge weights as electrical resistance, as defined by Ohm law [22]. Answer subgraphs with greater publication counts will be ranked higher, with publication counts derived from the curated knowledge sources treated with greater importance than those from the publication co-occurrences provided by OmniCorp. The confidence score is then augmented with an "informativeness" score, which is inspired by the NAGA scoring model [23] and treats novel more specific assertions (eg, disease X interacts with geneNPC1) with greater importance than more generic assertions (eg, disease X interacts with immune system).
The ROBOKOP KG can also be queried directly, independent of the UI, using the Cypher query language [24] to find subgraphs within the KG that match the structure of the query. Example Cypher queries can be found online [15].

Application Use Case Description
The EPR is a study of nearly 20,000 current participants that aims to better understand interactions among environmental exposures and genetic determinants of health and disease [25]. The registry contains survey data on participant exposures and disease history, as well as DNA samples and other biological measurements. As part of the broader effort, investigators have been exploring the impact of workplace exposures on IMDs.
An IMD was defined as a self-reported allergic reaction (allergic rhinitis, hay fever, or seasonal allergies; allergies [other than seasonal]); asthma condition; or autoimmune disorder (psoriasis, thyroid disease [noncancer], hyperthyroidism, hypothyroidism, Crohn disease, multiple sclerosis, celiac disease, Sjogren disease, rheumatoid arthritis, ulcerative colitis, scleroderma or systemic sclerosis, pernicious anemia, myositis, or lupus). EPR survey data were extracted in December 2018. The overall sample size was 4574 participants.
An exploratory analysis was conducted to examine associations between each IMD and specific workplace chemicals classified into one of 18 classes (Multimedia Appendix 2). The survey questions from which the chemicals were obtained were drawn from the EPR "Exposome Survey -Part A: External Exposome, Section B: Chemical and Metal Exposures at Work" and generally structured as follows: "Please select any heavy metals you have ever been exposed to for 15 minutes a week or more in any job you have held (CHOOSE ALL THAT APPLY)." Associations between workplace chemical exposures and individual IMDs were examined using chi-square analysis or the Fisher exact test, when sample sizes were small due to missing data or few positive cases. A false discovery rate correction was applied to the association tests. Chemicals were examined individually and also by chemical class. Odds ratios (ORs) with lower and upper bounds were calculated per convention and were not adjusted for small sample sizes. As this was an exploratory analysis, the significance level was set at α=.05 or .10, and we did not control for potential covariates such as age, sex, and race.

Application Use Case Results
A total of 45 exposure-IMD associations were significant at P<.10, with 20 associations significant at P<.05 after false discovery rate correction. In all cases, workplace chemical exposures were associated with increased odds of self-reported IMD (Multimedia Appendix 3). Dyes were the most common workplace exposure class associated with IMD conditions. No associations were identified between acids or glues/adhesives and IMD conditions. "Allergies or allergic reaction (other than seasonal allergies)" and "allergic rhinitis, hay fever, or seasonal allergies" were the most common IMD conditions associated with workplace chemical exposures.

ROBOKOP-Derived Mechanistic Assertions Based on Application Use Case Results
We highlight the ROBOKOP application using three exemplar associations that were chosen because they were significant at P<.05, evident at both the level of the specific chemical and the chemical class, and representative of different chemical classes and IMDs: (1) carbon monoxide-multiple sclerosis, (2) ammonia-asthma, and (3) isopropanol-allergic rhinitis, hay fever, or seasonal allergies. An overview of ROBOKOP results for each of these examples is provided below, with various functionalities of the interactive UI highlighted in the first example.

Carbon Monoxide and Multiple Sclerosis
The association between workplace exposure to carbon monoxide and multiple sclerosis was significant at both the chemical level (OR 6.4583, 95% CI 1.8524-18.2844; P=.006) and chemical class level (OR 3.8902, 95% CI 1.2521-10.3546; P=.03).
We posed the following question to ROBOKOP, but structured as a machine question: what genes might mediate the association between exposure to carbon monoxide and multiple sclerosis? The general question, machine question, and answer set are displayed in Figure 2. ROBOKOP identified seven subgraphs and potential intermediary genes as follows: TNF (tumor necrosis factor), BDNF (brain-derived neurotrophic factor), IL10 (interleukin-10), NGF (nerve growth factor), IRF8 (interferon regulatory factor 8), KCNMA1 (potassium calcium-activated channel subfamily M alpha 1), and CASP8 (caspase 8).
The top-ranked answer set had 858 PubMed publications, contributed by OmniCorp, supporting the association between multiple sclerosis and TNF, and 44 PubMed publications, again contributed by OmniCorp, supporting the association between carbon monoxide and TNF (Figure 3). Twenty-five additional PubMed publications (from OmniCorp) supported an association between carbon monoxide and multiple sclerosis, with several suggesting the involvement of heme oxygenase-1, which is described as an enzyme that oxidizes heme to bilirubin and carbon monoxide. The multiple sclerosis-TNF association was established by both HETIO and Pharos. The carbon monoxide-TNF association was established by CTD, with publication support contributed by CTD that again suggested a role for heme oxygenase-1 [26]. implemented as a machine question or meta-graph (B) designed to explore genes that might mediate the observed association between workplace exposure to carbon monoxide and Environmental Polymorphisms Registry participant self-report of multiple sclerosis. Users select nodes and edges in the ROBOKOP user interface (or by direct Cypher query) to translate the desired natural-language question into an executable machine question, using the schema provided in Figure 1 as a guide. The resultant aggregated answer graph (C) and list of answer subgraphs (D) show six potential mediating genes. Both the answer graph and the list of answer subgraphs are interactive and can be explored by users. For example in (D), users can click on each answer subgraph to explore knowledge sources, predicate assertions, and publication support. TNF: tumor necrosis factor; BDNF: brain-derived neurotrophic factor; IL10: interleukin-10; NGF: nerve growth factor; IRF8: interferon regulatory factor 8; KCNMA1: potassium calcium-activated channel subfamily M alpha 1; CASP8: caspase 8. Figure 3. The top-ranked answer subgraph suggesting the involvement of TNF (tumor necrosis factor) (A) as a potential gene that might mediate the observed significant association between workplace exposure to carbon monoxide and multiple sclerosis. The Reasoning Over Biomedical Objects linked in Knowledge Oriented Pathways (ROBOKOP) machine question was structured as follows: carbon monoxide -gene -multiple sclerosis. Example publication support for the linkage between carbon monoxide and TNF suggesting a role for heme oxygenase-1 (B). Metadata for the carbon monoxide node (C).

Ammonia and Asthma
The association between workplace exposure to ammonia and asthma was significant at both the chemical level (OR 2.0422, 95% CI 1.4426-2.8524; P=.001) and chemical class level (OR 1.5210, 95% CI 1.1938-1.9311; P=.004).
The lowest-ranked answer subgraph identified ADA as the intermediary gene. The metadata associated with ADA indicated that ADA metabolizes adenosine by converting it to inosine and ammonia. For this answer subgraph, the relationship between asthma and ammonia was supported by 93 PubMed publications, identified by OmniCorp. An ADA-asthma relationship was established using Monarch, with 32 supporting PubMed publications contributed by OmniCorp. Many of the supporting publications suggested a relationship between ADA mutations and asthma, allergy, and immune function [27].
As with the other two examples, the query that was posed to ROBOKOP aimed to identify intermediary genes and was structured similarly, except that "allergic disease" was used as the disease node, instead of "allergic rhinitis, hay fever, or seasonal allergies," because the latter disease node was not available in ROBOKOP.
ROBOKOP identified the following three intermediary genes: IL6, TNF, and CSF2 (colony stimulating factor 2). The intermediary gene in the second top-ranked answer set was IL6. An isopropanol-IL6 relationship was established by CTD, with one supporting publication provided by CTD and two additional supporting PubMed publications contributed by OmniCorp. An IL6-allergic disease relationship was established by Pharos, with 79 supporting PubMed publications contributed by OmniCorp. Many of those publications [28] suggested the involvement of the innate immune response (eg, basophils, eosinophils, and mast cells), and several publications suggested neuroimmune involvement (eg, "allergic mood" and hippocampal inflammation) [29].

Summary of Findings and Related Work
We identified 45 significant associations between workplace chemical exposures and IMD conditions in an EPR cohort, with 20 of those associations significant at P<.05 and largely unexpected a priori. Statistical evidence for an association, while important, does not establish a causal relationship or provide any insights into underlying mechanisms. Thus, we applied the open ROBOKOP KG system to validate the observed associations by demonstrating plausible connections between exposures and IMD conditions, and we provide mechanistic insights or hypotheses to explain them. We highlighted the following three use case applications: carbon monoxide-multiple sclerosis, ammonia-asthma, and isopropanol-allergic disease.
ROBOKOP identified plausible answer subgraphs to a query structured as carbon monoxide-gene-multiple sclerosis, thus supporting the statistical association between workplace exposure to carbon monoxide and multiple sclerosis. ROBOKOP further identified the gene that encodes TNF, TNF, as one of several potential mediating genes. ROBOKOP metadata and publication support suggested that heme oxygenase-1, which oxidizes heme to bilirubin and carbon monoxide, plays an intermediary role [26]. Levels of heme oxygenase-1 are depressed in patients with multiple sclerosis and further depressed during episodes of disease exacerbation [30], and exogenous carbon monoxide or chemicals that release carbon monoxide or induce heme oxygenase-1 appear to be therapeutic in experimental models of multiple sclerosis [31]. These results suggest that heme oxygenase-1, via carbon monoxide, has a homeostatic or anti-inflammatory role that might protect against multiple sclerosis or suppress disease exacerbation. However, more recent evidence suggests that a chronic heme oxygenase-1 response in glial cells may promote neurodegeneration and thereby exacerbate multiple sclerosis and other neurodegenerative diseases [32].
The statistical association between workplace exposure to ammonia and asthma was supported by multiple answer subgraphs that were returned by ROBOKOP. The gene that encodes ADA, ADA, was identified by ROBOKOP as one of several potential intermediates in this relationship. ROBOKOP metadata indicated that ADA metabolizes adenosine by converting it to inosine and ammonia. ROBOKOP identified publications suggesting an association between ADA mutations and asthma, allergy, and immune function, including one publication suggesting an association with aspirin-intolerant asthma [27]. A recent review indicated that ADA deficiency may have detrimental effects on multiple organ systems, including the pulmonary system [33]. Several older publications suggest that adenosine acts as a bronchoconstrictor in persons with asthma [34], and a more recent publication suggested an association between exposure to ammonia and asthma exacerbations [35]. These results support a relationship between ADA mutations and pulmonary complications, as well as an association between ammonia and pulmonary complications in persons with established respiratory disease.
ROBOKOP provided several answer subgraphs to support the statistical association between workplace exposure to isopropanol and allergic disease. The gene encoding IL6, IL6, was identified by ROBOKOP as one of several potential intermediary genes, with publication support suggesting the involvement of the innate immune response [28]. In addition, ROBOKOP identified publication support suggesting the involvement of IL6 in neuroimmune and neurobehavioral correlates of allergic disease [29]. While IL1 is widely recognized as playing a prominent role in "sickness behavior," IL6 also appears to play a role [36,37]. These findings suggest that the association between exposure to isopropanol and allergic disease might actually reflect a relationship between isopropanol and neurological/neurobehavioral correlates of allergic disease. Specifically, the results suggest that isopropanol might trigger an innate neuroimmune response that results in elevated levels of IL6, which then might trigger neurobehavioral symptoms of allergic disease.

Limitations
ROBOKOP has several limitations that should be considered when interpreting the results here or using the application. First, many of the associations between workplace chemical exposures and IMD conditions involved complex chemical mixtures such as "toner," "transmission fluid," and "motor oil." ROBOKOP currently maps these entities to Medical Subject Heading terms, but the application does not have an approach in place for mapping such mixtures to the individual chemicals that constitute a given mixture. We are considering approaches to overcome this limitation. Second, ROBOKOP, like all KG-based applications, is limited by the challenge of KG completion. For example, if an exposure-gene relationship has not been established by one of the curated data sources underlying the ROBOKOP KG, then this relationship will not be identified. We are developing algorithms to overcome this limitation by inferring edges in the KG, but for now, this remains a limitation. Third, evidence for ROBOKOP assertions is derived from the following two main sources: (1) the curated knowledge sources used to create the KG and (2) the co-occurrence of terms in PubMed abstracts. The ranking and scoring algorithm that is used to rank answer subgraphs is based on these two sources of evidence, with the first source treated with greater importance than the second. Other relevant factors, such as date of publication and number of studies on a topic (versus publications), are not considered at present, but may be incorporated into future versions of the application. Fourth, as a prototype system, ROBOKOP does not yet support natural language processing capabilities or other sophisticated approaches to aid users. We encourage users to contact the developers and/or post GitHub issues should they encounter any challenges when generating queries and/or evaluating answer subgraphs. Fifth, while not a limitation of ROBOKOP itself, the workplace chemical exposure-IMD associations reported here were derived from survey data (ie, participant self-report) and were not confirmed by clinical record review or expert judgement. As such, the potential exists for misclassification and/or bias in both the reported exposures and the reported IMDs. Moreover, the timing between workplace exposure and the onset of IMD cannot be determined due to limitations of the survey design and participant recollection at the time the survey was administered. Finally, as this was an exploratory analysis, we did not adjust the ORs for small sample sizes or control for potential covariates such as age, sex, and race.

Conclusion
ROBOKOP demonstrated potential in its use to support real-world observations and generate mechanistic hypotheses. In this paper, we focused on significant associations between workplace chemical exposures and IMDs, identified as part of a larger EPR study. We note, however, that ROBOKOP has other applications. Indeed, one key feature of ROBOKOP is its generalizability across biomedical domains as a general question-answering system, with capabilities to support a variety of machine questions. For instance, we are using ROBOKOP to explore associations between medications and clinical outcomes, including adverse events, using data derived from electronic health records. We are also testing whether ROBOKOP can be used to support human reasoning on Medical College Admission Tests, and we have promising preliminary results [38]. While the ROBOKOP KG is currently built from biomedical knowledge sources, the general approach is not restricted to the biomedical space. Indeed, on behalf of our institution's leadership, we are developing a new version of the ROBOKOP KG that is focused on exploring relationships between research proposals and investigator characteristics.
A second key feature of ROBOKOP is its ability to support speed to discovery. For instance, the example use cases presented here took little time to construct and execute, and the interactive UI allows even novice users to posit questions and explore answer subgraphs. The mechanistic insights that were gleaned from the ROBOKOP answer subgraphs were quickly realized, thus allowing for a rapid first-pass analysis of the results and evaluation of the supporting evidence. Moreover, ROBOKOP revealed all potential genes that might mediate the observed exposure-IMD associations via a single query; in effect, the user did not need to spend hours, days, or longer reading through the available literature. This speed to discovery afforded by ROBOKOP also allows investigators to quickly refute associations that may be nothing more than spurious findings (eg, if no answer subgraphs are returned by ROBOKOP). In the examples highlighted herein, we focus on three example exposure-IMD associations. While this may not seem like many, a full literature review to identify potentially novel insights, eliminate spurious findings, and explore supporting evidence would take abundantly more time than was needed to conduct the initial first-pass analysis using ROBOKOP. We plan to leverage ROBOKOP's speed to discovery in a large-scale analysis of associations between single nucleotide polymorphisms and phenotypes as part of a broader EPR effort and in several other studies.
A third key feature of ROBOKOP is its open nature. Indeed, access to ROBOKOP, whether via the UI or by direct Cypher query of the underlying KG, does not require login authentication or an account; rather, anyone with the URL can access the system. Moreover, the ROBOKOP KG can be downloaded independently of the application [15]. The open nature of ROBOKOP and the multiple routes to access it democratize science and, when coupled with the speed to discovery afforded by the application, should accelerate progress in biomedicine and many other fields.