extfieldsofstudy: Medicine Biology

text:
Discovering, Indexing and Interlinking Information Resources

The social media revolution is having a dramatic effect on the world of scientific publication. Scientists now publish their research interests, theories and outcomes across numerous channels, including personal blogs and other thematic web spaces where ideas, activities and partial results are discussed. Accordingly, information systems that facilitate access to scientific literature must learn to cope with this valuable and varied data, evolving to make this research easily discoverable and available to end users. In this paper we describe the incremental process of discovering web resources in the domain of agricultural science and technology. Making use of Linked Open Data methodologies, we interlink a wide array of custom-crawled resources with the AGRIS bibliographic database in order to enrich the user experience of the AGRIS website. We also discuss the SemaGrow Stack, a query federation and data integration infrastructure used to estimate the semantic distance between crawled web resources and AGRIS.

Introduction AGRIS (http://agris.fao.org/) is the International System for Agricultural Science and Technology, a collection of nearly 8 million multilingual bibliographic resources spanning the last forty years and produced by a network of more than 150 institutions from 65 countries. AGRIS is currently part of the CIARD initiative (http:// www.ciard.net/), a self-described "global movement dedicated to open agricultural knowledge". Some AGRIS data sources are unique (http://aims.fao.org/activity/blog/agris-enriched-data-fao) to the system and AGRIS is the only way in which they can be accessed. The system's goal is to make agricultural research globally discoverable, and as evidenced by Google Analytics it supports both developed and developing countries. Indeed, AGRIS is accessed from more than 200 countries and territories, reaching peaks of 250,000 visits per month. AGRIS users belong to two very different categories: the general public and agriculture professionals. In particular, a survey conducted at the end of 2014 helped to better describe the AGRIS audience [Celli et al., 2015]: researchers, professors, and graduate students looking for bibliographies, librarians, cataloguers, and people responsible for managing and disseminating research outcomes to the community and the rest of the world (including small and big journal publishers), and government officers asking for reports on specific topics. Since December 2013, AGRIS adopted a LOD (Linked Open Data) infrastructure [Anibaldi et al., 2015], which allowed the creation of mashup pages, where users looking for specific topics (e.g. impacts of climate change in a country) can access a publication from the AGRIS database, combined with other related resources extracted from other preselected datasets. External resources available in AGRIS mashup pages are not only bibliographic metadata, but also distribution maps, statistics, germplasm accessions, and so on. In this paper we explore a new data source available in AGRIS mashup pages: the web itself.
Nowadays, scientists and researchers publish their results not only in journals or at conferences, but also via web 2.0 tools and other media [Kouper, 2010;Shema et al., 2012] in order to efficiently and broadly communicate their outcomes; this technique also helps scientific research reach the general public, since newspapers, magazines and science blogs are often the quickest way to reach people informally. Blogs and other websites may also contain a corpora of ongoing research activities, unpublished material, grey literature, quick discussions, and experiments with negative results and ideas. The problem is that this information is usually not exposed using web services that can be consumed by machines, and the only way to access this rich amount of data is to use web search engines that typically return thousands of results, largely meaningless. In addition, most blogs and websites are not well categorized and so it is difficult for users and machines to discover what is actually relevant to the topic of interest.
In this context, we believe that it is important for AGRIS usersespecially for researchers -to have access to those valuable pieces of information that are neither exposed in a database nor accessible via web service. Our goal is to crawl the web, starting from a list of manually preselected websites and then apply a set of machine learning algorithms to categorize discovered web resources. In recent years, much research has been done to crawl and mine the web. Numerous solutions have been proposed [Devi et al., 2015;Liakos et al., 2015;Soulemane et al., 2012] to cope with the size of both the publicly indexable and the hidden web employing added semantics to discovered resources and to reuse them in fact sheets and mashups. In fact, it is not only important to discover web links, but also to process them in a way that allows reuse in multiple scenarios. The adoption of ontologies and LOD methodologies helps the analysis and enrichment of discovered web resources [Berendt et al., 2004]. Our work shows how it is possible to apply semantic enrichment to crawled web resources and to use this semantic knowledge to enhance the AGRIS web portal. More specifically, our work leverages Semantic Web technologies and the knowledge encoded in the AGROVOC (http://aims.fao.org/standards/agrovoc/about) thesaurus in order to recommend web resources that are relevant to a given AGRIS bibliographic item. AGROVOC is a multilingual vocabulary containing more than 32,000 agricultural concepts in 22 languages, aligned with 16 other multilingual knowledge organization systems related to agriculture, and developed by FAO over the course of thirty years [Caracciolo et al., 2011]. Adopting Semantic Web and LOD best practices and technologies, AGROVOC vocabulary items have been assigned URIs, organized into a SKOS-XL concept scheme (http://www.w3.org/TR/skos-reference/skos-xl.html), and served both as Linked Open data and via SPARQL endpoint (http://www.w3.org/TR/sparql11-overview).
In this article we discuss crawling and analysing web resources to populate our "Crawler Database"; a SPARQL endpoint with AGROVOC annotations of web resources identified by the URL from which they were crawled. By providing web resources with semantics we can use the AGROVOC descriptions of AGRIS bibliographic entries to interlink AGRIS and the Crawler Database. This linking is then exploited by a recommender that identifies web resources that are relevant to AGRIS entries. Furthermore, we also discuss the preliminary testing of the SemaGrow Stack (http://www. semagrow.eu/) as the computational infrastructure for interlinking the AGRIS bibliographic database with the Crawler Database. The query federation and data integration functionalities of the Sema-Grow Stack facilitate setting up experiments aiming at estimating semantic similarity between AGRIS entries and other resources.

Amendments from Version 1
We conducted an evaluation study on a benchmark sample of AGRIS articles in order to determine the relevance between crawled web resources and the AGRIS database. We computed the precision of recommendations considered as "relevant" by our algorithm, commenting on some possible improvements to the process used and described in our work. Outcomes of our evaluation study are presented in the "Analysis of relevance" section, together with a new picture displaying the cumulative distribution of AGRIS records over the number of relevant recommendations. Furthermore, we created a separate section "Analyzing the algorithm performance" where we compared the execution time of the recommender system in both the "individual" and "federated" modes. The section "The output of the recommender system" was removed, since it contained only a sample RDF/XML fragment that was not very significant. Lastly, the definition of the custom algorithm was removed and minor improvements have been made to the text, as suggested by reviewers.

REVISED
Although the core example that we discuss in this paper is based on the entities described in the Crawler Database, the power of the SemaGrow Stack is that it allows the re-use of this software in the context of different mashup pages combining AGRIS with a variety of LOD sources.
The entire process we discuss in this paper has already been implemented and integrated in the AGRIS website. While the tuning of the recommender system to compute accurate similarities is still an ongoing process, AGRIS mashup pages are enriched with the content of the Crawler Database and statistics are being collected in order to train the recommender system using user behaviour. In addition, the workflow and the components described in this paper can be used in any domain, so they are not restricted to agriculture; one can simply use another thesaurus to annotate web resources and populate the Crawler Database.

Crawling and indexing the web
The process of discovering and tagging web resources to display new content in the AGRIS website is based on two backend components: a customized Apache Nutch web crawler and Agro-Tagger. Figure 1 provides an overview of the entire process. As a starting point, in order to display relevant content in AGRIS, we manually preselect related websites to be used as input for the web crawler. Using these URLs the web crawler discovers other related web URLs, while AgroTagger assigns AGROVOC URIs to web URLs and creates the Crawler Database. In the next two sections we describe the two backend components.
The web crawler A web crawler is a piece of software that methodically and automatically analyses web pages provided as input. Each input web page is a ROOT of the crawling process. During the analysis of a web page, the web crawler discovers all the hyperlinks available in that page, adding them to the list of web pages to be visited. The process stops at a specific depth, i.e. the number of hops a discovered link is from the ROOT. The depth parameter is defined by the user of the web crawler, with the idea that links decrease in relevance as their distance from the ROOT grows. At the end of the crawling process, a list with discovered web URLs is produced. Figure 2 shows an example execution of the web crawler.
In order to implement the process of enriching the AGRIS website with relevant web resources, we used a customized version of Apache Nutch (http://nutch.apache.org/), a highly extensible, scalable and configurable open-source web crawler. Web URLs  provided as input were manually selected by experts in the domain of agriculture, in order to start the process from "trusted" and valuable websites. Currently, the depth parameter is set to 5, since this value seems to be a good compromise between discovering a good quantity of relevant resources and completing the task within an acceptable response time. The application, including the customized Apache Nutch web crawler and some Bash scripts to run it is available on GitHub (https://github.com/fcproj/agrotagger/tree/ master/crawler). The application requires three input parameters: -The depth of the crawling process -The path to the directory that stores the output of the process -The path to the text file that contains the list of web URLs used as ROOTs by the crawler The output of the application is a text file containing a structured list of discovered web URLs. Such file is built reading segments generated by the Apache Nutch crawler, using the "readsegdump" command line tool, as documented in the Apache Nutch documentation (http://wiki.apache.org/nutch/bin/nutch_readseg). The file structure is very easy to learn: a text file containing a URL per line; the URL can be specified after the "URL::" tag; otherwise, if the URL was discovered in an anchor, it can be specified in the "outlink: toUrl:" tag (if the anchor has a name, it is reported after the "anchor:" tag). Here is a snapshot of the output file:

AgroTagger classifier
AgroTagger engine (https://github.com/fcproj/agrotagger) is a toolkit that assigns semantic terms to textual content. At a high level of abstraction, it can be considered as a keyword extractor that uses the AGROVOC thesaurus to extract keywords from a set of web URLs. It is based on MAUI (https://code.google.com/p/mauiindexer), a tool that combines a keyphrase extraction algorithm and a machine learning toolkit for the identification of topics in textual documents. AgroTagger currently works only with English documents; in fact, AgroTagger is using the MAUI model trained with 780 full-text documents using AGROVOC in English [Medelyan & Witten, 2008]. Training MAUI with AGROVOC in other languages will allow the applicability of AgroTagger in a multilingual environment, even if further tests need to be performed to understand how the tool behaves with non-Latin characters. Regarding accuracy, a recent test (conducted outside the study proposed in this paper) in which AgroTagger results were analysed by professional indexers showed an accuracy of approximately 80%; in 20% of the cases the results were too broad. In brief, the accuracy measurement was carried out by domain experts actively involved in the development of the AGROVOC vocabulary based on manual annotations of a test sample from the AGRIS database. The test sample was composed of 32 documents already indexed by FAO professional cataloguers making use of the AGROVOC thesaurus; those documents were randomly selected from the AGRIS database, according with some constraints: the link to the full-text had to be available; they needed to be produced by FAO cataloguers; they needed to be indexed with at least 8 AGROVOC keywords. Then, AgroTagger was executed to automatically annotate such documents; lastly, keywords identified by cataloguers were compared with keywords assigned by AgroTagger, determining an accuracy of AgroTagger of around 80%.
In the process described in this paper, we apply AgroTagger to web URLs discovered by the web crawler and we annotate such URLs with AGROVOC URIs. Annotations are stored in a triple store (the Crawler Database) after which a recommender system defines some relevant combinations between AGRIS bibliographic resources and web documents, making use of AGROVOC as the backbone of the entire process. Considering AgroTagger as a black box (as depicted in Figure 3), we can describe its I/O as: -Input: web URLs discovered by the web crawler -Output: a set of triples that annotate web URLs AgroTagger is a multi-threaded application, guaranteeing better performance while manipulating web URLs. For each web URL available in the input file, AgroTagger: -Downloads the resource available at the given web URL and converts it to a text file -Runs the MAUI indexer trained with AGROVOC -Produces a set of annotations as RDF triples. The RDF schema of AgroTagger output annotations is shown in Figure 4)  Currently, AgroTagger produces an RDF NTRIPLE file that describes semantic annotations for web URLs. Here is an example of an annotated web URL: Training AgroTagger with a different thesaurus allows one to reuse the entire workflow and components described in this paper in completely different research domains.

The SemaGrow Stack
Scalable, efficient, and robust data services are re-shaping the way that data analysis techniques are applied to the heterogeneous data cloud and enable data-intensive and inter-disciplinary approaches to science. SemaGrow is an FP7 European project developing such infrastructure services. The core technical outcome of the project is the SemaGrow Stack. The SemaGrow Stack implements a SPARQL endpoint that federates SPARQL endpoints, transparently optimizing federated queries and dynamically integrating heterogeneous data models by applying the appropriate vocabulary transformations to queries and results [Charalambidis et al., 2015]. There are several key features of the SemaGrow Stack that address AGRIS use cases: it provides a querying interface that uses the result of ontology alignment to completely hide schema heterogeneity and also applies methods from database research and artificial intelligence that take into account data contents and optimize federated querying plans. The ontology alignment and dynamic vocabulary transformation facilities allow us to take advantage of multiple agricultural knowledge organization systems that have been aligned to AGROVOC. In this manner, one can develop AGROVOC-aware applications and use them over non-AGROVOC (but aligned) datasets without modification. The query optimizer is based on methods that automatically extract detailed metadata about the content of the federated endpoints, overcoming the lack of detail in manually provided annotations [Zoulis et al., 2015]. As an added benefit, the SemaGrow Stack implements fail-over mechanisms that fall back to alternatives in the face endpoint unavailability. It furthermore does not require any modification of the federated endpoints or any other obtrusion of current workflows.
The SemaGrow Stack is developed and distributed as open-source software (https://github.com/semagrow) and requires an Apache Tomcat environment (http://tomcat.apache.org/) in order to be deployed and executed.

The recommender system
The Crawler Database is composed of triples generated by Agro-Tagger. At this stage, the biggest problem is to compute a meaningful intersection with the AGRIS bibliographic database in order to display relevant information in an AGRIS mashup page. First of all we need to define the concept of "meaningful combinations". A naïve approach is based on counting the number of AGROVOC URIs in common between resources from the Crawler Database and resources from the AGRIS bibliographic database. Thus, for an AGRIS mashup page, we can state that we want to display those resources from the Crawler Database having as many AGROVOC URIs as possible in common with the AGRIS bibliographic entry.
To implement this naïve algorithm we developed a recommender system (https://github.com/fcproj/recommender), a JAVA component that computes meaningful combinations between the Crawler Database and the AGRIS database, and generates a new triplestore: the "Recommender Database". The recommender system runs as required as new data is periodically generated by the web crawler.
The recommender system can make use of the SemaGrow Stack as the backbone of the process, as depicted in Figure 5: in this way, the recommender system is able to compute meaningful combinations between all datasets federated by SemaGrow. Combinations are computed by counting the number of common "dcterms:subject" URIs (http://dublincore.org/documents/dcmi-terms/) between entities of the federated datasets (in the case of the AGRIS dataset, "dcterms:subject" URIs are AGROVOC URIs). The SemaGrow Stack is not strictly necessary, since the recommender system can also work querying two target SPARQL endpoints one by one. In any case, the SemaGrow Stack allows code reuse for all datasets; it is sufficient to configure the SemaGrow Stack by defining the URLs of the SPARQL endpoints included in the federation, and the recommender system can work using a single endpoint. Without the usage of the SemaGrow Stack as an intermediate layer, there is the need to modify the code to add further SPARQL endpoints to the process. Let's explain this statement with an example. Currently, the recommender system has two ways of working (the user can configure it to use either mode): "federated", which requires a single SPARQL endpoint, and "individual", which accepts two SPARQL endpoints. The "federated" mode makes use of the SemaGrow Stack: if the user configures the Stack defining three or four endpoints in the federation, the recommender system can combine all of them. On the contrary, the "individual" mode works with a maximum of two SPARQL endpoints and would require additional software code if one wanted to add further endpoints to the process.
Our experiments focus on the usage of the recommender system to interlink the AGRIS database and the Crawler Database, storing computed combinations in the Recommender Database. At this stage, the algorithm is very easy: we simply need to count the number of common AGROVOC URIs (expressed as objects of a "dct:subject" predicate) between entities of the AGRIS dataset and the Crawler Database. Using the SemaGrow Stack and the single query "federated" mode, the SPARQL query to implement the algorithm is: This SPARQL query sorts resources from the Crawler Database by the number of AGROVOC URIs in common with a given AGRIS entity <$AGRIS_URI> after which it takes the 20 most relevant web resources and stores them in the Recommender Database as a set of recommendations for an AGRIS URI. In the query, the statement including the rdf:type predicate is only needed in order to get results from the Crawler Database; since the recommender system can work with any dataset federated by the SemaGrow Stack, the type needs to be an application-dependent parameter so that it can be configured to query different datasets.
Unfortunately, there are still few limitations with the SemaGrow Stack. First of all, SPARQL queries must be optimized and they are affected by their content. For instance, using a FILTER statement makes for a high response time. Thus, in order to identify entities from the different datasets federated by the SemaGrow Stack, it is necessary that such datasets define an "rdf:type" (http://www. w3.org/TR/rdf-schema/) for their entities in order to avoid the usage of the FILTER statement. Moreover, given that at the time of these experiments the SemaGrow Stack distribution was limited and allowed only SPARQL queries with dct:subject and/or rdf: type predicates, we ran our experiments using the "individual" mode during processing and testing. At the time of publication, our final experiments using the "federated" mode are very promising; SPARQL queries that combine triples from federated datasets show a response time comparable to the execution time of a process that runs individual SPARQL queries to different endpoints and programmatically combines their results.
The "individual" mode that makes use of two different SPARQL queries to two different SPARQL endpoints: AGRIS and the Crawler Database. The first SPARQL query provides the list of AGROVOC URIs ("dct:subject" predicate) for a given <$AGRIS_URI>: Then, a custom algorithm implemented in JAVA is used to count the number of common AGROVOC URIs between the AGRIS entity and web URLs returned by the previous query, storing the top 20 most relevant web resources in the Recommender Database.

The similarity index
In the Recommender Database, recommendations for the same AGRIS URI are sorted by relevance (currently, the relevance is given by the custom algorithm calculated using the number of AGROVOC URIs in common with the AGRIS resource). The recommender system provides a Similarity Score for each recommended web URL, i.e. the percentage of similarity between an AGRIS resource and a recommended web URL. There are three variables to take into account in order to determine the Similarity Score: -the number of AGROVOC URIs associated with an AGRIS record (we will refer to this variable as #AGRIS); -the number of AGROVOC URIs associated with a recommended web URL (#WEB) and -the number of common AGROVOC URIs between the web resource and the AGRIS record (#COMMON).
A naïve approach would consider the Similarity Score as the ratio given by the division between #COMMON and #AGRIS: Unfortunately, this approach has some problems when the AGRIS record is associated with few AGROVOC URIs. For instance, consider the scenario described in Table 1.
The final two scores have an obvious problem of overestimation: the AGRIS record is associated with only 3 AGROVOC URIs and, even if the web URL has 3 common URIs with the AGRIS record, a score of 1.0 (100%) is too much to predict the similarity. Thus, we need an index to adjust the score when the AGRIS record has few AGROVOC URIs associated. This Similarity Index must be 1 in the case of maximum similarity, and 0 in the case of no common AGROVOC URIs. A naïve Similarity Index would be: The problem with this index is that # WEB is equal to 10 in most cases, since AgroTagger assigns 10 AGROVOC URIs to web resources crawled by the web crawler. Thus, S = 1 only if #AGRIS is 10 and #COMMON is 10, but this situation is highly improbable, since #AGRIS is on average equal to 6. If we define a threshold τ to determine when a number of common AGROVOC URIs is relevant to determine a good Similarity Index, we can improve the quality of the Similarity Score. Currently, we have defined τ equal to 6, which means that all cases where #COMMON is equal or bigger than 6, the Similarity Index must be 1.0. In this way, defining the correction factor k: The Similarity Index σ can be computed as: This index has all the properties we are looking for: if #COMMON is 0, σ is 0; if #COMMON is equals or bigger than the threshold τ (6), σ is 1. At this point, we can redefine the Similarity Score as: This approach further improves the Similarity Score. Since #WEB is never bigger than 10, while #AGRIS could be ideally any positive integer, we can modify the denominator of the first factor introducing the upper limit T, which is the maximum number of AGROVOC URIs associated to an AGRIS resource that is meaningful to the computation of the Similarity Score (we define T = 10): Where: T = 10 and τ = 6. Considerations about the custom algorithm. The custom algorithm to compute meaningful combinations is an area for further research and improvement. In particular, other parameters may play a key role to define the relevance of a web resource. For instance, the system might check if some AGROVOC terms used by an AGRIS record appear in the title of the crawled web resource, or in its description. Then, AgroTagger could also be improved to return the ranking score of each AGROVOC URI assigned to a web resource, in order to use such a score in the algorithm that computes combinations. In addition to that, we can observe users' behaviour in an AGRIS mashup page in order to assign more relevance to web resources more frequently clicked by AGRIS users. Furthermore, interviews of users will be conducted to evaluate the relevance of resources displayed in AGRIS mashup pages. Finally, there may be other more exotic algorithms to calculate similarity that can give greater relevance to end users, but such experiments were outside the scope of this project.
Analyzing the algorithm performance The algorithm was executed on 775,297 AGRIS URIs; the crawler database was composed of 17,320,363 triples; the computation generated 19,738,145 triples for the recommender triple store.
The recommender system ran in a CentOS 5 environment with the following configuration: -Processor: Intel(R) Core(TM) i7 2.80GHz We ran two experiments. In a first experiment we used the Sema-Grow Stack as the backbone of the process, so the recommender system worked in the "federated" mode. For each AGRIS URI, the computation of combinations required 1.89 seconds on average. The execution time range was between 1.4 and 3.02 seconds for each AGRIS URI. Differences depended on three aspects: -the number of AGROVOC URIs for a given AGRIS URI, which affects the response time of the AGRIS SPARQL endpoint. On average, an AGRIS resource contained 6 AGRO-VOC URIs; -the specificity of an AGROVOC concept; broader concepts are associated to many web URLs, so they affect the response time of the Crawler Database SPARQL endpoint and -network speed and delays.
It is important to note that this is a background, offline process that runs periodically to keep the database of recommendations up to date. End users querying AGRIS resources are served from this database, and are insulated from the processing time of the recommendation system. In this context, the aforementioned response times are perfectly acceptable as they allow for very frequent (daily or even more frequent) updates using moderate computational resources.
In a second experiment we ran the recommender system in the "individual" mode. The algorithm was executed on the same resources as for the previous experiment. For each AGRIS URI, the recommender system needed 2.3 seconds on average. Thus, using the recommender system as intermediate layer reduced the execution time of around 0.4 seconds for each AGRIS URI.

The AGRIS front-end
The creation of an AGRIS mashup page (an example id provided in Figure 6) is a dynamic and sensitive process, which is made possible by the usage of AGROVOC as the backbone of the system. In fact, AGRIS records come with AGROVOC URIs and, relying on AGROVOC formal alignments to many thesauri, it is possible to query publicly accessible web services or SRARQL endpoints to provide access to resources indexed with various thesauri. Thus, when the user selects a publication from the AGRIS database, the system can display related information on the same topic. External data sources are identified based on the content, the relevancy to the AGRIS domain, and the information provider [Celli et al., 2015].  A new data source: related resources crawled from the web This paper presented a set of components that implement a process of discovering web resources related to AGRIS records in order to enrich the user experience in AGRIS mashup pages. We also ran experiments with the SemaGrow Stack as a backend component that can easily extend the data sources federated by the AGRIS mashup pages. These experiments led directly to the addition of a new data source to the AGRIS mashup pages: the dataset of related resources crawled from the web.
The creation of this new data source required setting up a process that used numerous components. We implemented an automatic process that makes use of a web crawler to discover web resources, and relies on AgroTagger to annotate discovered URLs with AGROVOC URIs. Then, in order to compute meaningful combinations between the AGRIS database and the Crawler Database, we implemented a recommender system to define the relevance of web resources. The output of this process is a widget in the AGRIS mashup pages that, reading the content of the Recommender Database (that is continuously updated by an offline process), can display relevant resources from the web. In this way, we believe that AGRIS users may find relevant data that assists them in working with agricultural issues and food security.

Analysis of relevance
In this section we describe an evaluation study conducted on a benchmark sample of AGRIS articles, in order to determine the relevance of resources contained in the widget of Figure 7. We used a set of 10 AGRIS articles in the domain of fisheries and the relevance was evaluated by domain experts. Criteria to determine the test set were: -AGRIS articles must be in the domain of fisheries (this is because we could rely on experts in the fisheries area). To cope with this requirement, we ran a query to the AGRIS Solr index, which is the index used by the website to allow end users to look for articles available in AGRIS; we used a Boolean "OR" query based on around 100 terms in the fisheries domain (including general keywords like "Fish", "Fisheries", "aquaculture", but also more specific concepts like "Lumpfish", "John dory", "Sea bass", "Whitefish", "Weevers", "Dolly varden" and "Carp").
-AGRIS articles must be indexed with a number of AGRO-VOC keywords ranging from 4 to 8. In fact, as discussed in the "Similarity Index" section, while AgroTagger assigns 10 keywords to a crawled web resource, AGRIS record can contain on average 6 AGROVOC terms, ranging from 0 to 14. In our analysis we wanted to exclude extreme situations. We are obliged to make a consideration; there are two main strategies when a cataloguer indexes some articles: the cataloguer may use very few specific terms, or a lot of terms including also more general ones. Choosing to avoid extreme situations, we tried to balance between AGRIS records with too many general terms and records with very few specific terms. The mean of AGROVOC keywords per record in the AGRIS database was used as a pivot for our decision.
-10 AGRIS articles were randomly selected by the subset of AGRIS articles meeting the previous two requirements.
Evaluating 10 AGRIS records means manually evaluating 50 recommendations coming from the recommender system. In fact, for each AGRIS record we recommend 5 web crawled resources in the widget "Activities from the web". The evaluation followed a precise workflow: -for each AGRIS article in the test set, we had to understand the topic of the article, considering the title, the abstract and the AGROVOC keywords available.
-for each recommended web resource in the widget "Activities from the web", we identified if it was relevant to the AGRIS article or not. We paid special attention to the Similarity Score and to the level of granularity of AGROVOC keywords assigned to the AGRIS article.
Recommended web resources could have been identified as "not relevant" in some specific cases: -they were considered generic news/publication pages, too vague to be relevant; -the resource was geographically relevant, but the specific matter was completely different; -the resource was related to a completely different subject; -the resource was related to a too generic subject; this case was mainly due to the lack of specificity of AGROVOC keywords attached to an AGRIS article.
An important remark is that recommendations about the same specific topic of the AGRIS article were considered relevant even if they referred to a different geographical region. Moreover, we found one recommendation pointing to a web resource no more available (HTTP 404 response code). As an example, consider the AGRIS article "Monitoring and Surveing Fingerling Releasing (Kutum Bream and Pike Pearch) in Quality and Quantity Point of View", indexed with 6 AGROVOC keywords. One of the keywords is "fisheries", which is may be too generic; in fact, the fourth recommendation (having a Similarity Score of 50%) is not relevant to the article because, even if it is geographically relevant, it talks about "fisheries", while the AGRIS article is about "aquaculture techniques". Note that the first three recommendations are all relevant (i.e. the precision is 0.6) and they present a Similarity Score between 50% and 66.67%. Figure 8 displays the percentage of AGRIS records in the benchmark sample with a minimum number of relevant recommended resources; the histogram points out that at least one recommended resource per AGRIS article was considered relevant. The analysis of results showed a general precision of 0.52 (26 relevant resources over 50 analysed recommendations). For each AGRIS record in the test set, precision ranged from 0.2 to 0.8. 39% of recommendations considered as "not relevant" were too vague or related to a very general subject. Thus, some improvements to the algorithm of the recommender system can be made to get a better precision. The analysis also indicated that a Similarity Score bigger than 50% showed an acceptable level of precision: 27 recommendations had a Similarity Score greater than or equal to 50%, but one of them pointed to a non-existing web page; 18 recommendations were considered relevant, with a precision of 0.66 (0.69 if we exclude the recommendation no more available).
An interesting example can demonstrate how the algorithm implemented by the recommender system can be improved by matching AGROVOC terms of the AGRIS article with the titles and descriptions of crawled web resources. Let's consider an extreme situation, i.e. an AGRIS article with many AGROVOC keywords, including also more generic ones. The article "Water Conservation of Qanat, using Optimum use of Water in Unused Seasons (Case study: Dehraz Qanat of Sabzevar)" exposes some criteria to optimize the use of water carried by qanats (a qanat is an underground channel to transport water from an aquifer under a hill, especially for irrigation of hot and arid environments). Thus, as it is highlighted by the title, the article is mainly about "water conservation". It has been indexed with eleven AGROVOC terms, including water management, soil sciences, soil conservation, and water conservation. The recommender system suggests five web resources related to this article, with a similarity score ranging between 42% and 60%. The first suggestion is a web page of the USDATA.GOV website (http:// afsic.nal.usda.gov/soil-and-water-management/water-conservation); the score is 60% and the resource is completely related with the article, since it contains other resources about "water conservation" coming from USA universities and other agencies. Matching AGROVOC terms of the AGRIS article with the title of this resource would have further increased the Similarity Score, since the title itself is about "water conservation". Regarding the remaining four suggestions, the example shows how a similarity score of 42% does not predict good results. For instance, the web page of Cornell University (http://conservationagriculture.mannlib.cornell. edu/pages/resources/photosvideos.html) is relevant to the article, since it contains -among other things -some Power Point presentations about "water management" and "soil management". Conversely, the other resources with the same similarity score of 42% are irrelevant. The improved algorithm (matching AGROVOC terms

Minimum number of relevant recommendations Percentage of AGRIS records
of the AGRIS article with the titles and descriptions of crawled web resources) would suggest the XML feeds from EPRINTS about "agricultural water management", and the PDF document entitled "Agricultural Perspectives on Water Resource Management in the Americas", both of which are quite relevant and yet missing from the current list. In this case, the improved algorithm would increase the precision from 0.4 to 0.8.

Conclusions
The web contains much latent knowledge, especially when that knowledge is expressed as unstructured and poorly categorized full-text content. This paper describes a proposed solution to discover such knowledge making use of modified open source software (Nutch and Maui) together with the SemaGrow Stack and a custom recommender in order to enrich the relevance of AGRIS bibliographic resources and hence the AGRIS web portal mashup.
The adoption of the SemaGrow Stack as a backend facilitated the development of a recommender engine as it was possible to implement the system without requiring any prior knowledge of the specifics of the datasets that are combined with the AGRIS database. In this manner, the system can be re-used with any dataset using AGROVOC (or any terminology aligned to AGROVOC) to describe websites, experiments, software, or any resources relevant to agriculture. We are now able to show to AGRIS users any relevant content extracted from the web, something possible thanks to the adoption of semantic web technologies. The entire process will continue to be extended and improved as experiments continue: AgroTagger and the recommender system will be tuned to guarantee better precision in the computation of recommendations (for instance, trying to match AGROVOC keywords with titles of target resources and experimenting with alternative algorithms), while the SemaGrow Stack will be optimized and used to federate additional data sources.
Furthermore, as previously mentioned, training AgroTagger with a different thesaurus allows to apply the entire process to different research domains, and not only to the AGRIS website.

Software availability
Latest source code (AgroTagger): https://github.com/fcproj/agrotagger Latest source code ( Author contributions FC conceived conceptual ideas, implemented all the software components (except for the SemaGrow Stack), designed the experiments, and prepared the manuscript.
JK made provided guidance to the development of the AGRIS website, as well as conceptual suggestions to the implementation of the crawling and tagging processes.
YJ participated in AGRIS modelling, study design and selection of technology solutions. He also prepared and revised the language of the manuscript. SK contributed sections 3 and part of section 1. He was also responsible for the conceptual and technical development of the Sema-Grow Stack.
DV supported the testing of the crawling and recommendation processes, providing access to computing resources of PARADOX supercomputing facility at the Scientific Computing Laboratory of the Institute of Physics Belgrade.
We want to thank again Dr. Missier for his comments. As far as the title concerns, we decided to make it very general, since the workflow and components that we describe in the paper can be adapted to all domains. It is true that our use case (i.e. AGRIS) refers to the domain of agricultural science and technology. Anyway, as we discussed in the paper, what makes the workflow domain-specific is the AgroTagger; if someone trains such tool with a SKOS thesaurus in a different domain, he/she can apply the workflow to the new use case. It is also true that there are other restrictions; for instance, you should use a thesaurus available as SKOS; you should be prepared to work with RDF and SPARQL, since the output of the recommender system is a set of triples, as well as its input is a set of SPARQL endpoints. But the entire workflow is applicable to the entire web.
About the code, it is very easy to be learned and used. Documentation is provided within the code, and the software components are highly configurable (you simply need to change a configuration file and then to execute a JAVA class). The only difficult part is training the AgroTagger with a new thesaurus; in fact, it requires to build a new MAUI model, which needs a medium-large set of documents already indexed with the specific thesaurus by humans.
No competing interests were disclosed.

Competing Interests:
23 November 2015 Referee Report doi:10.5256/f1000research.7931.r11244 Kei Kurakawa National Institute of Informatics, Tokyo, Japan The paper has been modified to take my suggestions into consideration. The authors strongly took care with a more delicate explanation for the quality of proposed functionality, i.e., recommendations of web resources for an AGRIS article. They also revised the whole description of their thoughts on the paper in an acceptable way. This paper, as a software tool article, is acceptable to be indexed.

I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.
No competing interests were disclosed. Current study entitled "Discovering, Indexing and Interlinking Information Resources" consists of a Current study entitled "Discovering, Indexing and Interlinking Information Resources" consists of a technical evaluation of an enriching system attached to an agricultural bibliographic database called AGRIS. I am not sure about the novelty of the system that is presented but I can admit that is an interesting work which does not pose any reading difficulties to the reader, even to those who are not so familiar. Lately, recommending systems have been set on the center of the research activity and the specific work is following the research trends of the area.
In my opinion there are some gaps in the study. For example on the description of the Agrotagger crawler researchers say: "Regarding accuracy, a recent test in which AgroTagger results were analyzed by professional indexers showed an accuracy of approximately 80%; in 20% of the cases the results were too broad. In brief, the accuracy measurement was carried out by domain experts actively involved in the development of the AGROVOC vocabulary based on manual annotations of a test sample from the AGRIS database" It is not clear to me if the aforementioned test has been held in the context of another previous study or it covers the specific study. If it has been held in the past a reference is necessary. Otherwise researchers need to provide further details. Additionally I think that authors should provide more information about accuracy tests. In my opinion the paper describes adequately the design and implementation but needs further enrichment regarding evaluation data.
Authors should consider papers restructuring as in many cases I get lost between implementation and evaluation points.
Some minor syntax comments are: In Page 1: "AGRIS database, other related resources extracted from other preselected together with "… "together with" should be changed to "combined with". datasets In Page 1: "Nowadays, scientists and researchers publish their results not only in journals or at conferences, but also via web 2.0 tools and other media [ ;Kouper, 2010 ] in order to Shema et al., 2012 efficiently and broadly communicate their outcomes, [a technique that also helps scientific research reach the general public, since newspapers, magazines and science blogs are often the quickest way to reach For better understanding the sentence should be split into two. From the point "a people informally]." technique' should start another sentence.
In Page 1 "Corpus" should be : "Blogs and other websites may also contain a corpus of ongoing …" changed into "corpora" In Page 2: "Adopting Semantic Web and LOD best practices and technologies, AGROVOC vocabulary items have been assigned URIs, organized into a SKOS-XL concept scheme ( ), and served both as Linked Open data and via http:  Re-write the sentence for better SPARQL endpoint ( )." http://www.w3.org/TR/sparql11-overview understanding.
I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.
No competing interests were disclosed.

Competing Interests:
No competing interests were disclosed.

Competing Interests:
Author Response 03 Nov 2015 , Food and Agriculture Organization of the United Nations, Italy

Fabrizio Celli
Really thanks for your comments. As you suggested, the "AgroTagger" section was improved. We added a reference to a paper including details about the MAUI training set. We also added a more precise description of the test sample used to measure accuracy of AgroTagger. In addition to that, evaluation of performance was moved to a separate section, together with further details as the comparison between the execution time in the individual and federated modes. Moreover, we improved the evaluation of relevance of recommendations: we ran an analysis of relevance on a test set of resources in the domain of fisheries, computing precision and providing other comments. We hope that, based on these modifications, the referee can now fully approve our article.
No competing interests were disclosed. This paper proposed a functional enhancement for the agricultural journal article bibliographic database search, AGRIS, whose bibliographic page has additional web resources information relevant to its content. The web resources are automatically crawled by a customized Nutch web crawler, and labeled with some agricultural index terms, i.e. AGROVOC by a kind of machine learning module, the AgroTagger classifier. Since AGRIS bibliographies are already labeled with AGROVOC, the paper proposed a similarity measure between AGRIS bibliographies and web resources based on the number of common and originally labeled AGROVOC terms on them. AGRIS bibliographies and web resources are stored in SPARQL servers, so that the SemaGrow stack is used to easily send queries to them to calculate the similarity scores. The recommender system provides at most top 20 relevant web resources for an AGRIS bibliography by the calculation, which is serialized on RDF/XML. I recognize that the whole framework to construct the new functionality is valuable, but the paper is not sufficient with computational performance evaluation on the framework and quality evaluation for the linkages between bibliographies and web resources in an approved scientific method.
Below are the suggestions for minor improvements.
In page 3: In the "Crawling and indexing the web" section, the author describes "In the section 2.1 and 2.2" in the leading paragraph. The section numbers might not be forbidden to point to sections of this article.
In page 4: In the left column of the "web crawler" sub-section, the author shows a snap shot of the output file of a Nutch web clawer. Please describe the structure of the file. I would like to know what some tags mean to In the left column of the "web crawler" sub-section, the author shows a snap shot of the output file of a Nutch web clawer. Please describe the structure of the file. I would like to know what some tags mean to construct the structure. Why is a blank URL listed in the example?
In the left column of the "AggroTagger classifier" sub-section, the author mentions the quality of tagging by AgroTagger such as "AgroTagger results were analyzed by professional indexers showed an accuracy of approximately 80%; in 20% of the cases the results were too broad." I would like to know more precise descriptions of the quality of tagging. How large is the training set? How did the author construct the training set? Please describe the training set. In addition, the test set for tagging and examining its accuracy should be described.
In the "AggroTagger classifier" sub-section, the figure 3 and 4 should be explicitly referenced. In the current article, those figures are not referenced in any sentences.
In the first phrase of "The SemaGrow Stack" section, the author refers "the data-intensive and inter-disciplinary Science of 2020". Does "2020" means "Horizon 2020"? Or, does it refer to another funding framework? Science of 2020? If it is another funding framework, please cite a relevant web site for it. Please make sure what it is.
In page 6: In the "recommender system" section, the second paragraph of "The recommender system can make use of … add further endpoints to the process", the figure 5 should be explicitly referenced. In addition, for good understanding of reconfigurable structures consisting of the "federated" and the "individual", the author had better depict both structures in the figure 5.
In page 7: In the right column, the author describes "the overall custom algorithm" as a list form. For better understanding, this algorithm should be represented in a pseudo-programming code form.
In the right column, the author mentions some experimental conditions and results of the number of AGRIS URIs and the average execution time. In addition, how many AGROVOC URIs are prepared for AGRIS URIs or Web URLs in this experiment? How long does it take for all processing? Please figure out the time of processing in the "individual" mode and the "federated" mode. Which is better from the viewpoint of computation?
In page 8: In the right column, line 13, "factor K" should be changed to the lower case "factor k".
In the last similarity formula, "braces ( )" should be written in the proper place.
In table 2, "Real Score" might be " Revised Score"?
In page 9: In the right column, the section "AGRIS front-end", please indicate explicitly figure 6 in a sentence.
I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.
No competing interests were disclosed. We would really like to thank Dr. Kei Kurakawa for this useful review. The paper was improved following his comments. In particular: We separated the performance experiments from the deployment section. Performance experiments were performed again using a bigger test set and we compared execution time in the "individual" mode with the execution time in the "federated" mode.
We ran an analysis of relevance on a test set of resources in the domain of fisheries. We were able to compute precision of linkages, adding some additional details.
The "web crawler" section was revised: we described the structure of the output file, which is generated using the "readseg -dump" command line tool provided by Apache Nutch. The blank URL listed in the example was meaningless, so we removed it. The "AgroTagger" section was revised too, with the addition of a more precise description of the test sample used to measure accuracy. Then, the paper including the work of training MAUI was cited. In the "SemaGrow Stack" section, the reference to the "Science of 2020" was removed, since it was misleading. In fact, it is part of SemaGrow proposal description, so it does not explicitly refer to Horizon 2020 or to any other framework, but to a general view of the world in 2020. All minor improvements suggested by the reviewer were implemented. We hope that, based on these modifications, the referee can now fully approve our article.
No competing interests were disclosed.

Paolo Missier
School of Computing Science, Newcastle University, Newcastle upon Tyne, UK Overall, I found the article interesting and easy to read. In line with the F1000 reviewers' guidelines, I am going to comment "on the quality of the research and whether the article is scientifically sound (i.e. whether the work has been well designed, executed and discussed)", rather than on novelty, also making suggestions for improvements in the presentation.
The work partly builds on prior established tools designed for the AGRIS system, mainly the AGROVOC thesaurus, as well as methods for federating SPARQL queries over heterogeneous resources (the SemaGRow stack).
The original research contribution consists mainly of a focused crawler that looks for web resources that are relevant to the AGRIS community, the AgroTagger annotation tool, which annotates those web resources with semantic terms from the AGROVOC thesaurus, and a recommender tool that uses the annotated database to suggest web resources that are likely to be relevant to a given AGRIS bibliographic item.
The annotation model is fine, however the evaluation part of the paper is weak, especially with regards to The annotation model is fine, however the evaluation part of the paper is weak, especially with regards to recommendation. This is where I think the authors have an obligation to report on the accuracy (ie F-scores) of their method on a benchmark of test cases, especially since substantial effort is spent in the paper to explain the similarity score used by the recommender. This is completely missing, however, the only evidence of the method at work being an example towards the end. For an experimental paper, this is hardly acceptable. So my main substantial recommendation is to strengthen the evaluation section (this may appear in other articles cited in the references, however that was not immediately clear to me).
On matters of form and presentation, I found that the methods, including the recommender algorithm, are very simple (by the authors' own admission) and thus in my opinion the exposition can be greatly abbreviated. For example, fig. 2 and 3 are not very informative and could be removed. Also the lengthy RDF fragments are not very significant (but if you need them, please switch to n3 notation!), the annotation model of fig 4 is very simple (essentially, just a dcterms:subject), the 6-points algorithm is a straightforward intersection operation and has already been described in the text, the queries on pg 7 are fairly simple, etc.

Further minor presentation suggestions:
You may not need to build up the similarity score from naive to final, possibly just describe the final version?
There is a mix of design and implementation / performance considerations, these may be best kept separate.
A new section header may be missing after the first paragaph of "agris front end"?
I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.
No competing interests were disclosed.

Competing Interests:
Author Response 03 Nov 2015 , Food and Agriculture Organization of the United Nations, Italy

Fabrizio Celli
We would really like to thank Dr. Paolo Missier for his useful comments.
We ran an analysis of relevance on a test set of resources in the domain of fisheries. We were able to compute precision, but not recall. In fact, recall is mainly related to the web crawling, and it is not easy to estimate it. In addition to that, in the context of the AGRIS website it is not strongly important: we are more interested in a good precision, especially because we display only 5 recommendations per AGRIS resource.
We separated the performance experiments from the deployment section, adding further details. Performance experiments were performed again using a bigger test set and we compared execution time in the "individual" mode with the execution time in the "federated" mode.
We abbreviated a bit the exposition (for instance, we removed the six-points algorithm and the RDF fragment related to the output of the recommender system), but we maintained the discussion about the flow of thoughts leading from the naïve similarity score to the final one because we thought it helps in justifying some parameters available in the final formula.


Joint Event Trigger Identification and Event Coreference Resolution with Structured Perceptron

Events and their coreference offer useful semantic and discourse resources. We show that the semantic and discourse aspects of events interact with each other. However, traditional approaches addressed event extraction and event coreference resolution either separately or sequentially, which limits their interactions. This paper proposes a document-level structured learning model that simultaneously identifies event triggers and resolves event coreference. We demonstrate that the joint model outperforms a pipelined model by 6.9 BLANC F1 and 1.8 CoNLL F1 points in event coreference resolution using a corpus in the biology domain.

Introduction
Events convey semantic information such as who did what to whom where and when. They also corefer to each other, playing a role of discourse connection points to form a coherent story. These aspects of events have been already utilized in a wide variety of natural language processing (NLP) applications, such as automated population of knowledge bases (Ji and Grishman, 2011), topic detection and tracking (Allan, 2002), question answering (Bikel and Castelli, 2008), text summarization (Li et al., 2006), and contradiction detection (de Marneffe et al., 2008). This fact illustrates the importance of event extraction and event coreference resolution.
Those semantic and discourse aspects of events are not independent from each other, and in fact often work in interactive manners. We give two examples of the interactions: (1) British bank Barclays had agreed to buy(E1) Spanish rival Banco Zaragozano for 1.14 billion euros. The combination(E2) of the banking operations of Barclays Spain and Zaragozano will bring together two complementary businesses.
(2) The Palestinian Authority condemned the attack(E3), saying it(E4) would divert international sympathy away from the far higher Palestinian civilian death toll.
E1 corefers to E2, and E3 does to E4. E2 is more abstract than E1, and has less evidence of being an event. E4 is a pronoun, and thus may seem to refer to an entity rather than an event. Thus, E2 and E4 are relatively difficult to be recognized as events by themselves. However, event coreference E1-E2, which is supported primarily by E2's participants Barclays and Zaragozano shared with E1, helps determine that E2 is an event. The same logic applies to E3 and E4. On the other hand, previous works typically rely on a pipelined model that extracts events (e.g., E1 and E3) at the first stage, and then resolves event coreference at the second stage. Although this modularity is preferable from development perspectives, the pipelined model limits the interactions. That is, the first stage alone is unlikely to detect E2 and E4 as events due to the difficulties described above. These missing events make it impossible for the second stage to resolve event coreference E1-E2 and E3-E4. In this work, we address the problem using the ProcessBank corpus (Berant et al., 2014). Following the terminology defined in the corpus, we introduce several terms: • Event: an abstract representation of a change of state, independent from particular texts. • Event trigger: main word(s) in text, typically a verb or a noun that most clearly expresses an event. • Event arguments: participants or attributes in text, typically nouns, that are involved in an event. • Event mention: a clause in text that describes an event, and includes both a trigger and arguments. • Event coreference: a linguistic phenomenon that two event mentions refer to the same event.
We aim to explore the interactions between event mentions and event coreference. As a first step toward the goal, we focus on the task of identifying event triggers and resolving event coreference, and propose a document-level joint learning model using structured perceptron (Collins, 2002) that simultaneously predicts them. Our assumption is that the joint model is able to capture the interactions between event triggers and event coreference adequately, and such comprehensive decision improves the system performance. For instance, the joint model is likely to extract E2 as well as E1 successfully via their event coreference by simultaneously looking at coreference features. Our contributions are as follows: 1. This is the first work that simultaneously predicts event triggers and event coreference using a single joint model. At the core of the model is a document-level structured perceptron algorithm that learns event triggers and event coreference jointly. 2. The incremental token-based prediction in joint decoding poses a challenge of synchronizing the assignments of event triggers and coreference. To avoid this problem, we propose an incremental decoding algorithm that combines the segment-based decoding and best-first clustering algorithm. 3. Our experiments indicate that the joint model achieves a substantial performance gain in event coreference resolution with a corpus in the biology domain, as compared to a pipelined model.

Related Work
No previous work deals with event extraction and event coreference resolution simultaneously. We thus describe how these two tasks have been addressed separately, and how joint structured learning has been studied in other NLP tasks. Event extraction has been studied mainly in the newswire domain and the biomedical domain as the task of detecting event triggers and determining their event types and arguments. In the former domain, most work took a pipelined approach where local classifiers identify triggers first, and then detect arguments (Ji and Grishman, 2008;Liao and Grishman, 2010;Hong et al., 2011). Li et al. (2013) presented a structured perceptron model to detect triggers and arguments jointly. Similarly, joint dependencies in events were also addressed in the latter domain (Poon and Vanderwende, 2010;McClosky et al., 2011;Riedel and McCallum, 2011;Venugopal et al., 2014). However, none of them incorporated event coreference into their model.
Event coreference resolution is more challenging and less explored. To set up event triggers as a starting point of the task, some works use human annotation in a corpus (Bejan and Harabagiu, 2014;Liu et al., 2014), and others use the output of a separate event extraction system (Lee et al., 2012). Berant et al. (2014) presented a model that jointly predicts event arguments and event coreference (as well as other relations between event triggers). However, none of them tries to predict event triggers and event coreference jointly.
Joint structured learning has been applied to several NLP tasks, such as word segmentation and part-of-speech (POS) tagging (Zhang and Clark, 2008a), POS tagging and dependency parsing (Bohnet and Nivre, 2012), dependency parsing and semantic role labeling (Johansson and Nugues, 2008), the extraction of event triggers and arguments (Li et al., 2013), and the extraction of entity mentions and relations (Li and Ji, 2014). Their underlying ideas are similar to ours. That is, one can train a structured learning model to globally capture the interactions between two relevant tasks via a certain kind of structure, while making predictions specifically for these respective tasks. However, no prior work has studied the interactions between event trigger identification and event coreference resolution.

Approach
We formalize the extraction of event triggers and event coreference as a problem of structured prediction. The output structure is a document-level event graph where each node represents an event trigger, and each edge represents an event coreference link between two event triggers.

Corpus
The ProcessBank corpus consists of 200 paragraphs from the textbook Biology (Campbell and Reece, 2005). Table 1 shows statistics of our data splits. The original corpus provides 150 paragraphs as training data, and we split them into 120 and 30 for our training and development, respectively. We chose ProcessBank instead of a larger corpus such as the Automatic Content Extraction (ACE) 2005 corpus for the following two reasons. First, the human annotation of event coreference links in ProcessBank enables us to apply the bestfirst clustering directly; on the other hand, this is not readily feasible in ACE 2005 since it annotates event coreference as clusters, and gold standard event coreference links required for the bestfirst clustering are not available. Second, event coreference resolution using ProcessBank is novel since almost no previous work on the task used that corpus. The only exception could be (Berant et al., 2014), where they extracted several types of relations between event triggers, including event coreference. However, they did not report any performance scores of their system specifically on event coreference, and thus their work is not comparable to ours.  Unlike previous work (Berant et al., 2014;Li et al., 2013), we explicitly allow an event trigger to have multiple tokens, such as verb phrase 'look into' and compound proper noun 'World War II'. This is a more realistic setting for event trigger identification since in general there are a considerable number of multi-token event triggers 1 .

Event Graph Learning
Let x denote an input document with n tokens where x i is the i-th token in the document. For event graph learning, we use structured perceptron (Collins, 2002), and average weights to reduce overfitting as suggested in (Collins, 2002). The algorithm involves decoding to generate the best event graph for each input document. We elaborate on our decoding algorithm in Section 3.3. Since an event graph has an exponentially large search space, we use beam search to approximate exact inference. We extract a range of features by using Stanford CoreNLP , MATE (Björkelund et al., 2009), OpenNLP 2 , Nomlex (Macleod et al., 1998), and Levin verb classes (Levin, 1993). For brevity, we provide details of the structured perceptron algorithm and features in the supplementary material.
We use the standard-update strategy in our structured perceptron model. As variants of structured perceptron, one could employ the early up-date (Collins and Roark, 2004) and max-violation update (Huang et al., 2012) to our model. Our initial experiments indicated that early updates happen too early to gain sufficient feedback on weights from entire documents in training examples, ending up with a poorer performance than the standard update. This contrasts with the fact that the early-update strategy was successfully applied to other NLP tasks such as constituent parsing (Collins and Roark, 2004) and dependency parsing (Zhang and Clark, 2008b). The main reason why the early update fell short of the standard update in our setting is that joint event trigger identification and event coreference resolution is a much more difficult task since they require more complex knowledge and argument structures. Due to the difficultly of the task, it is also very difficult to develop such an effective feature set that beam search can explore the search space of an entire document thoroughly with early updates. This observation follows (Björkelund and Kuhn, 2014) on entity coreference resolution. In contrast, the maxviolation update showed almost the same performance as the standard update on the development data. From these results, we chose the standardupdate strategy for simplicity.

Joint Decoding
Given that an event trigger has one or more tokens, event trigger identification could be solved as a token-level sequential labeling problem with BIO or BILOU scheme in the same way as named entity recognition (Ratinov and Roth, 2009). If one uses this approach, a beam state may represent a partial assignment of an event trigger. However, event coreference can be explored only from complete assignments of an event trigger. Thus, one would need to synchronize the search process of event coreference by comparing event coreferences from the complete assignment at a certain position with those from complete assignments at following positions. This makes it complicated to implement the formalization of token-level sequential labeling for joint decoding in our task. One possible way to avoid this problem is to extract event trigger candidates with a preference on high recall first, and then search event coreference from those candidates, regarding them as complete assignments of an event trigger. This recalloriented pre-filtering is often used in entity coreference resolution (Lee et al., 2013;Björkelund Algorithm  and Farkas, 2012). In our initial experiments, we observed that our rule-based filter gained around 97% recall, but extracted around 12,400 false positives against 823 true positives in the training data. This made it difficult for our structured perceptron to learn event triggers, which underperformed on event coreference resolution.
We, therefore, employ segment-based decoding with multiple-beam search (Zhang and Clark, 2008a;Li and Ji, 2014) for event trigger identification, and combine it with the best-first clustering (Ng and Cardie, 2002) for event coreference resolution in document-level joint decoding. The key idea of segment-based decoding with multiple-beam search is to keep previous beam states available, and use them to form segments from previous positions to the current position. Let l max denote the upper bound on the number of tokens in one event trigger. The k-best partial structures (event subgraphs) in beam B at the j-th token is computed as follows: where 1 ≤ l ≤ l max , y [1:j] is an event subgraph ending at the j-th token, and y [j−l+1,j] = s means that partial structure y [j−l+1,j] is a segment, i.e., an event trigger candidate with a subsequence of tokens x [j−l+1,j] . This approximates Viterbi decoding with beam search. The best-first clustering incrementally makes coreference decisions by selecting the most likely antecedent for each trigger. Our joint decoding algorithm makes use of the incremental process to combine the segment-based decoding and bestfirst clustering. Algorithm 1 shows the summary of the joint decoding algorithm. Line 3 -7 implements the segment-based decoding, and line 8 -11 implements the best-first clustering. Once a new event trigger is appended to an event subgraph at line 6, the decoder uses it as a referring mention regardless of whether the event subgraph is in the beam, and seeks the best antecedent for it. This enables the joint model to make a more global decision on event trigger identification and event coreference decision, as described in Section 1.

Experimental Settings
When training our model, we observed that 20iteration training almost reached convergence, and thus we set the number of iterations to 20. We set l max to 6 because we observed that the longest event trigger in the entire ProcessBank corpus has six tokens. When tuning beam width k on the development set, large beam width did not give us a significant performance difference. We attribute this result to the small size of the development data. In particular, the development data has only 28 event coreferences, which makes it difficult to reveal the effect of beam width. We thus set k to 1 in our experiments.

Baseline Systems
Our baseline is a pipelined model that divides the event trigger decoding and event coreference decoding in Algorithm 1 into two separate stages. It uses the same structured perceptron with the same hyperparameters and feature templates. We choose this baseline because it clearly reveals the effectiveness of the joint model by focusing only on the architectural difference. One could develop other baseline systems. One of them is a deterministic sieve-based approach by Lee et al. (2013). A natural extension to the approach for performing event trigger identification as well as event coreference resolution would be to develop additional sieves to classify singletons into real event triggers or spurious ones. We leave it for future work.

Evaluation
We evaluate our system using a reference implementation of coreference scoring algorithms . As for event trigger identification, this scorer computes precision (P), recall (R), and the F1 score. With respect to event coreference resolution, the scorer computes MUC (Vilain et al., 1995), B 3 (Bagga and Baldwin, 1998), two CEAF metrics CEAF m and CEAF e (Luo, 2005)  BLANC (Recasens and Hovy, 2011) extended by . We also report the CoNLL average (Denis and Baldridge, 2009), which is the average of MUC F1, B 3 F1, and CEAF e F1.

Results and Discussions
We first show the result of event coreference resolution on the test data in Table 2. The joint model outperforms the baseline by 6.9 BLANC F1 and 1.8 CoNLL F1 points. We observed that this overall performance gain comes largely from a precision gain, more specifically, substantially reduced false positives. We explain the superiority of the joint model as follows. In the baseline, the second stage uses the output of the first stage. Since event triggers are fixed at this point, the baseline explores coreference links only between these event triggers. In contrast, the joint model seeks event triggers and event coreference simultaneously, and thus it explores a larger number of false positives in the search process, thereby learning to penalize false positives more adequately than the baseline.  Table 3: Results of event trigger identification. 'Baseline' refers to the first stage of our baseline. Table 3 shows the results of event trigger identification on the test data. We observed that the joint model also reduced false positives, similarly in event coreference resolution. However, its improvement on precision is small, ending up with almost the same F1 point as the baseline. We speculate that this is due to the small size of the corpus, and the joint model was unable to show its advantages in event trigger identification.
Below are two error cases in event coreference resolution, where our model fails to resolve E5-E6 and E7-E8. The model was unable to adequately extract features for both event triggers and event coreference, particularly because their surface strings are not present in training data, they are lexically and syntactically different, and they do not share key semantic roles (e.g., agents and patients) in a clear argument structure.
(4) The next seven steps decompose(E7) the citrate back to oxaloacetate. It is this regeneration(E8) of oxaloacetate that makes this process a cycle.

Conclusion and Future Work
We present a joint structured prediction model for event trigger identification and event coreference resolution. To our knowledge, this is the first work that solves these two tasks simultaneously. Our experiment shows that the proposed method effectively penalizes false positives in joint search, thereby outperforming a pipelined model substantially in event coreference resolution.
There are a number of avenues for future work. One can further ensure the advantage of the joint model using a larger corpus. Our preliminary experiment on the ACE 2005 corpus shows that due to its larger document size and event types, one will need to reduce training time by a distributed learning algorithm such as mini-batches (Zhao and Huang, 2013). Another future work is to incorporate other components of events into the model. These include event types, event arguments, and other relations such as subevents. One could leverage them as other learning targets or constraints, and investigate further benefits of joint modeling.


extfieldsofstudy:
Computer Science
Mathematics

s2fieldsofstudy:
Computer Science
Business

text:
Frequency Spreading Equalization in Multicarrier Massive MIMO

Application of filter bank multicarrier (FBMC) as an effective method for signaling over massive MIMO channels has been recently proposed. This paper further expands the application of FBMC to massive MIMO by applying frequency spreading equalization (FSE) to these channels. FSE allows us to achieve a more accurate equalization. Hence, higher number of bits per symbol can be transmitted and the bandwidth of each subcarrier can be widened. Widening the bandwidth of each subcarrier leads to (i) higher bandwidth efficiency; (ii) lower complexity; (iii) lower sensitivity to carrier frequency offset (CFO); (iv) reduced peak-to-average power ratio (PAPR); and (iv) reduced latency. All these appealing advantages have a direct impact on the digital as well as analog circuitry that is needed for the system implementation. In this paper, we develop the mathematical formulation of the minimum mean square error (MMSE) FSE for massive MIMO systems. This analysis guides us to decide on the number of subcarriers that will be sufficient for practical channel models.

I. INTRODUCTION
Based on the recent discussions on the 5 th generation of wireless communication networks (5G), massive MIMO, as a candidate, can play a pivotal role in these networks, [1]. Additionally, as a fundamental building block in the physical layer of 5G networks, the need for signaling techniques that are compatible with the emerging applications such as the machine to machine (M2M) communications and Internet of Things (IoT) is greatly emphasized [2]. Filter bank multicarrier (FBMC), a method that was initially proposed about 50 years ago, [3], [4], is recently being considered as a candidate for 5G systems, [1].
Massive MIMO is a code division multiple access (CDMA) like system that may be used to increase the capacity of multiuser networks, [5]. As it is shown in [6], by increasing the number of antennas at the base station (BS), the effects of noise and multiuser interference (MUI) start to vanish until they will be completely removed as the number of BS antennas tends to infinity. In the recent works [7] and [8], FBMC was proven to be a good match for massive MIMO and vice versa as they can both bring a great number of appealing properties into the picture of 5G systems. From network capacity viewpoint, this combination is of a paramount importance as the same spectrum is not only being reused by all the users (an advantage of massive MIMO) but it is being used in a more bandwidth efficient way.
A few recent publications have introduced an alternative to the traditional polyphase-based implementation of the FBMC systems that is termed as frequency spreading FBMC (FS-FBMC), [9]- [13]. This technique offers a number of advantages that put FBMC in a strong position as a candidate for the future wireless standards. FS-FBMC was explained for the first time in a document that came out of the PHYDYAS project, [9]. Further details of FS-FBMC were presented, by the same author, in [10]. The sensitivity of FS-FBMC to the timing offset is studied in [11] where the robustness of the FS-FBMC frequency domain equalizer to the timing errors compared with its counterpart with a polyphase-based receiver structure was reported. Berg et al., [12], have explored the utilization of FS-FBMC to access TV white spaces. All these works have used the filter designed by Martin [14] and Mirabbasi and Martin [15] for the implementation of FS-FBMC system.
As an extension to our previous work, [7], in this paper, we introduce the frequency spreading equalization (FSE) concept to the realm of FBMC-based massive MIMO systems. Thanks to the better equalization capability of FS-FBMC systems, we will show that FSE brings a great amount of performance enhancement compared with the results of [7]. This allows the usage of even wider subcarriers than what was suggested in [7] which in turn further reduces the latency due to the synthesis and analysis filter banks. Furthermore, this unlocks utilization of higher order modulation schemes in FBMC-based massive MIMO systems. Using a smaller number of subcarriers leads to a lower complexity, lower sensitivity to CFO, reduced PAPR and reduced latency. These advantages simplify the digital and analog circuitry that is needed for the system implementation.
Widening the bandwidth of each subcarrier is equivalent to shortening the length of the prototype filter. This, in turn, has the impact of increasing the bandwidth efficiency of FBMC, as the ramp-up and ramp-down of the FBMC signal will be shortened. A factor that limits utilization of FBMC systems in some applications like bursty M2M communications is the long ramp-up and ramp-down of their signals. To tackle this problem, some researchers have introduced the concept of FBMC systems with circular pulse-shaping, [16]- [19], where the prototype filter transients can be removed through the so called tail biting property. Even though these systems can remove the prototype filter transients from their signals, they need to add a cyclic extension to form their signal. Additionally, they suffer from higher out of band emissions than the FBMC systems with linear pulse-shaping, [18], [20]. The approach in this paper is to reduce the ramp-up and rampdown of the FBMC signal through widening the subcarriers while achieving a good equalization performance. This re-duced transient duration of the linearly pulse-shaped FBMC signals is equivalent to the cyclic extensions being used to form the circularly pulse-shaped FBMC signals. Therefore, in terms of bandwidth efficiency, both systems seem to be about the same while linearly pulse-shaped FBMC systems keep the advantage of having a lower out of band emissions.
The rest of the paper is organized as follows. FBMC signal construction and the frequency spreading method are explained in Section II. The proposed frequency spreading equalization for the massive MIMO systems is discussed in Section III where we have also derived analytical signal to interference plus noise (SINR) expressions. Various aspects of our design are analyzed through computer simulations in Section IV. Finally, the paper is concluded in Section V.
Notations: Scalars, vectors and matrices are represented in regular letters, boldface lower and upper case letters, respectively. I N denotes an N × N identity matrix. The matrix or vector superscripts (·) T and (·) H indicate transpose and conjugate transpose, respectively. · and | · | represent Euclidean norm and absolute value, respectively. ℜ{·}, ℑ{·}, E[·] and (·) −1 identify the real and imaginary parts of a complex number, expectation and inverse of a matrix, respectively. Linear convolution is denoted by ⋆, and the superscript * represents the complex conjugate.

II. CMT AND ITS FREQUENCY SPREADING IMPLEMENTATION
In cosine modulated multitone (CMT), which is a type of FBMC similar to the widely known OQAM-FBMC technique [21], real-valued data symbols are spread across time and frequency such that if the symbol time spacing, i.e., symbol period, is denoted by T , the symbol frequency spacing is F = 1 2T . Moreover, subcarrier phases are toggled between 0 and π/2 so that adjacent subcarriers have a quadrature phase difference. This phase-adjustment is necessary for the CMT receiver to be able to recover the transmitted data symbols free of interference along time and frequency. The filter p(t), which is used to pulse shape the data symbols, is a squareroot Nyquist filter, i.e., q(t) = p(t) ⋆ p * (−t) is a Nyquist filter, where the spacing between its zero-crossings is equal to 2T .
In practice, the CMT signal is efficiently synthesized at the transmitter and analyzed at the receiver using the polyphase network (PPN) [22], [23]. However, as it has been recently suggested by Bellanger [9], the frequency spreading structure can be utilized to gain a number of advantages compared with the traditional PPN method. In particular, an interesting property of FS-FBMC structure is its high-performance equalization capability. This equalization, namely, FSE, can be utilized in the context of massive MIMO as suggested in Section III. In the rest of this section, and for our later reference, we review the frequency spreading structure when applied to CMT, which we hereafter call FS-CMT.
The basic idea of the frequency spreading method is that a discrete-time square-root Nyquist filter, p[n], of size N = KL, can be synthesized in the frequency domain using a set of 2K − 1 distinct tones, as Here, c k is a real-valued coefficient that indicates the frequency response of the p(n) at the frequency ω k = 2πk N , K is the overlapping factor, i.e., it signifies the number of symbols that overlap in the time domain and L is the sample spacing of the zero-crossings of the Nyquist filter q(n) = p(n) ⋆ p * (−n).
When the above idea is extended to the multicarrier case, the symbols at each subcarrier should be pulse-shaped in the frequency domain using the set of c k 's. An inverse discrete Fourier transform (IDFT) block of size N can then be used to calculate the time domain samples from their frequency domain counterparts. Casting the above discussion into the mathematical form, the FS-CMT symbol can be obtained as Here, s(n) = s 0 (n) s 1 (n) · · · s L−1 (n) T , where s k (n) indicates the data symbol at the k th subcarrier and n th symbol time, Φ indicates the phase-adjustment matrix which is an L × L diagonal matrix with the diagonal elements {1, e j π 2 , . . . e j π 2 (L−1) }, A is the spreading matrix of size N × L whose k th column contains the coefficients c ℓ 's that are centered at the center frequency of the k th subcarrier, F H N indicates the IDFT matrix and x(n) is an N × 1 vector indicating the n th FS-CMT symbol in time domain.
Noting that the size of each FS-CMT symbol is N and the symbol spacing is equal to L/2, the transmit signal can be constructed by performing the overlap-and-add operations on the symbol vectors as illustrated in Fig. 1. Assuming an ideal channel and having a perfect synchronization between transmitter and receiver, the N samples that correspond to the n th symbol are collected in the vectorr(n), at the receiver side, and the data symbols are recovered according tô Since we assumed an ideal channel, equalization was not considered here. In the presence of the channel, the equalization can be embedded after obtaining the DFT output samples, [10].
In the next section, we extend the equalization scheme that was originally proposed for single antenna systems in [10] to the context of massive MIMO.

III. MMSE-FSE IN MASSIVE MIMO
An interesting property of FBMC in massive MIMO channels that is called self-equalization was recently reported in [7]. Due to that property, a channel flattening effect takes place as a result of linear combining of the channel gains in different receiver antennas during the equalization procedure. As a result, wider subcarriers in frequency can be utilized which brings a number of benefits to FBMC systems in massive MIMO application. The benefits of FBMC-based massive MIMO systems are well articulated in [7] and [8].
As it is shown in [11], a better channel equalization can be achieved through frequency spreading implementation of FBMC systems compared with the FBMC implementations based on polyphase networks. Hence, based on the results of [11] and [7], we are interested in the investigation of the FS-FBMC in massive MIMO channels. As will be shown in Section IV, due to the higher frequency resolution of FS-FBMC, the subcarrier bands in these systems can be widened even more than what is reported in [7]. Thus, the latency due to the synthesis and analysis filter banks can be reduced to a great extent compared with single antenna systems.
In this section, we derive the mathematical foundation for MMSE-FSE in massive MIMO systems. In a multiuser MIMO system similar to the one in [6], let N r be the number of receive antennas at the base station and M the number of users. Each user's mobile terminal (MT) is equipped with a single transmit/receive antenna. All MTs are simultaneously communicating with the BS in time division duplexing (TDD) mode. The number of BS antennas is considered much larger than the number of users, i.e., N r ≫ M . The users communicate with the BS using CMT modulation. The users' signals are distinguished by their respective channel gains between each MT antenna and the BS antennas. The channel gain vectors for different users are assumed to be independent with respect to each other. Recalling the structure of the CMT receiver from Section II, the output at the i th bin of the FFT block from the ℓ th user at all the receive antennas can be stacked into an N r × 1 vectorr ℓ i = r ℓ i h i (ℓ) where h i (ℓ) is the channel gain vector whose elements are the gains between the ℓ th MT antenna and the BS antennas at the i th frequency bin. The coefficient r ℓ i is the received signal at the i th bin of the output of the FFT block at the receiver from user ℓ in an ideal channel. The vectorsr ℓ i are the received signals from different users that contribute to form the received signal at the BS. Accordingly, the received signal vector can be written as where ] T and v i is an N r × 1 vector that contains the FFT samples of the additive white Gaussian noise (AWGN) at the i th bin of the output of the FFT blocks at different receive antennas.
In order to estimate different users' transmitted symbols, a number of equalization techniques can be utilized; namely, matched filter (MF) and MMSE. However, as it is noted in [7], in realistic situations when the number of BS antennas is finite, MF equalization incurs some performance degradation due to the residual multiuser interference. In this paper, our discussion will be limited to the MMSE-FSE which is known to be a superior technique.
In MMSE-FSE for massive MIMO systems, similar to single antenna FS-FBMC systems, equalization needs to be performed before despreading the output of the FFT blocks. Therefore, the MMSE linear combining aims at estimating the vector r i that contains the equalized samples of different users at the output of their FFT blocks. The MMSE solution of (4) is optimal as it maximizes the signal-to-interference-plus-noise (SINR), [24]. Following the same line of derivations as in [7], the MMSE estimates of r i 's can be obtained aŝ where the coefficient matrix contains the optimal MMSE filter tap weights for different users in its columns. It is worth mentioning that the elements of the noise vector v i are assumed to be independent and identically distributed Gaussian random variables with variances of After the MMSE estimation of the vectorsr i for i = 0, . . . , N − 1, frequency despreading can be separately applied for each user. To this end, letR = [r 0 , . . . ,r N −1 ] T . Consequently, the MMSE estimates of the transmitted data symbols of different users, i.e., s ℓ for ℓ = 0, . . . , M −1 can be obtained through frequency despreadingŜ where the L × M matrixŜ contains the MMSE estimation of the transmitted data symbols of different users, i.e.,ŝ ℓ 's, on its columns. Based on equations (4) to (6) and the assumption of having a flat fading channel gain over each frequency bin of DFT, the output SINR of the CMT receiver with MMSE channel equalization for the ℓ th user at subcarrier i can be calculated as follows.
where P (ℓ,i) s and P (ℓ,i) I are signal and interference-plus-noise powers, respectively. P (ℓ,i) s and P (ℓ,i) I can be obtained using equations (8) and (9) that are shown on the top of the previous page where w i,ℜ (m) and h i,ℜ (m) represent the real parts of the m th columns of the matrices W i and H i , respectively. Similarly, the subscript ℑ in the equations (8) and (9) shows the imaginary part of the corresponding vectors. It is worth mentioning that equation (7) is used as a benchmark to investigate the channel flatness assumption in Section IV.

IV. NUMERICAL RESULTS
In this section, the theoretical developments of the paper are analyzed and corroborated through numerical results. We use the results of [7] as the basis in order to evaluate the signal processing power of the FS method in the context of massive MIMO. It is worth noting that all of our simulations are based on a sample set of channel responses generated according to the SUI-4 channel model proposed by the IEEE802.16 broadband wireless access working group, [25]. Additionally, the channels between different users and different antennas are considered independent with respect to each other.
In the first set of simulations (Fig. 2), we study the effect of increasing the number of antennas on the signal-to-interference ratio (SIR) performance. In Fig. 2, a single-user in a noisefree channel is considered to investigate the self-equalization property in FS-FBMC based massive MIMO systems. To evaluate the robustness of the FSE to the subcarrier width, we have repeated the simulations for three cases of 8, 16, and 32 subcarriers and different number of receive antennas at the base station. More specifically, for the total bandwidth of 2.8 MHz, we consider the subcarrier widths of 350, 175, and 87.5 kHz, respectively. SIRs are evaluated at all the subcarrier channels. Note that in each curve, the number of points along the normalized frequency axis is equal to the number of subcarriers, L. As these results highlight, we can achieve a significant performance improvement even for the subcarrier spacings as large as 350 kHz.
The effectiveness of these results will be more clear if we compare the FSE technique with the conventional PPN algorithm. Fig. 3 compares the SIR performance of the PPN-FBMC with FS-FBMC system. A single-tap equalizer per subcarrier is used in the PPN-FBMC structure. From Fig. 3, it can be understood that the SIR improvement of higher than 30 dB can be achieved through FS-FBMC in comparison with Next, we consider a multiuser scenario with M = 6 users, L = 16 subcarriers and K = 4. With the assumption of having perfect power control for all the users, the signal-to-noise-ratio (SNR) of SNR in = −1 dB at the input of BS antennas is considered. The output SINR can be obtained using SNR in + 10 log 10 N r in dB. Fig. 4 compares the SINR values from simulation for two cases of N r = 128 and N r = 64 with the theoretical SINR values that are calculated based on the assumption of having a flat fading channel per subcarrier band that is calculated from equation (7). As the figure depicts, the theoretical derivations match perfectly with the simulation results. This emphasizes that the channel is flattened over all the subcarrier bands, thanks to the self-equalization property. The subcarrier spacing that achieves the same SINR as that of (7) is 4 times larger than what was suggested in [7] which brings a significant improvement in terms of shortening the length of the prototype filter and hence reducing the latency that is imposed by the transmit and receive filter banks.

V. CONCLUSION
In this paper, the idea of frequency spreading FBMC was extended to the massive MIMO systems. In addition, an effective MMSE equalization scheme for FBMC-based massive MIMO systems was derived. It was shown that further subcarrier widening than what was suggested in [7] is possible using FS-FBMC structure while gaining from self-equalization property. Based on the SIR and SINR performance results, higher order modulation techniques can be utilized in FS-FBMC based massive MIMO systems. Furthermore, improvements in terms of bandwidth efficiency, robustness to frequency offsets, complexity, PAPR and latency compared with the polyphase based FBMC systems can be achieved using FS-FBMC structure.