Learning to Recommend Items to Wikidata Editors

Wikidata is an open knowledge graph built by a global community of volunteers. As it advances in scale, it faces substantial challenges around editor engagement. These challenges are in terms of both attracting new editors to keep up with the sheer amount of work and retaining existing editors. Experience from other online communities and peer-production systems, including Wikipedia, suggests that personalised recommendations could help, especially newcomers, who are sometimes unsure about how to contribute best to an ongoing effort. For this reason, we propose a recommender system WikidataRec for Wikidata items. The system uses a hybrid of content-based and collaborative filtering techniques to rank items for editors relying on both item features and item-editor previous interaction. A neural network, named a neural mixture of representations, is designed to learn fine weights for the combination of item-based representations and optimize them with editor-based representation by item-editor interaction. To facilitate further research in this space, we also create two benchmark datasets, a general-purpose one with 220,000 editors responsible for 14 million interactions with 4 million items and a second one focusing on the contributions of more than 8,000 more active editors. We perform an offline evaluation of the system on both datasets with promising results. Our code and datasets are available at https://github.com/WikidataRec-developer/Wikidata_Recommender.

Introduction
Wikidata is an open knowledge graph built by a global community of volunteers [40]. Since its launch in 2012 it reached more than 90 million items and 24, 000 active editors . Manual contributions are core to Wikidata [40]: editors add new content, keep it up-to-date, model knowledge as graph items and properties, and decide on all rules of content creation and management. However, as Wikidata advances in scale, it faces substantial challenges around editor engagement [34].
Experience from other online communities and peer-production systems, including Wikipedia, Quora and others, suggests that one way to improve engagement is through recommendations [7,9,44]. This is especially true for editors who are relatively new to Wikidata, who need to overcome so-called 'legitimate peripheral participation' (LPP) effects to continue to engage [28]. According to LPP, newcomers are more likely to become members of a community if they are provided with suggestions of (typically lower-risk) tasks they could carry out to further the goals of the community [19]. At the moment, looking for relevant items to edit in Wikidata remains challenging because of the sheer number of options available [34]. The suggested and open tasks page gives a useful but daunting overview of the various ways in which people could contribute to Wikidata. Many task lists are automatically generated, without taking into account aspects such as editor tenure, previous editing history or interests. This lack of focus is also said to prevent editors from developing reinforced editing habits, which increases the likelihood of dropouts [34]. Seasoned editors use tools such as QuickStatements and Watchlist to organise their work, but such tools become relevant only once editors have identified productive ways to contribute.
For these reasons, we propose WikidataRec, a recommender system for Wikidata items. The system uses content-based and collaborative filtering techniques to rank items for editors relying on both item features and item-editor previous interaction. Collaborative filtering is a representative approach to address recommendation task with implicit feedback, which learns item-centric and editor-centric edit representations using matrix factorization [17] from item-editor interaction data. We adopt it in this work and further facilitate it with additional information from item content and relations, where we learn these representations from by sentence embedding model ELMo [26] and graph embedding model TransR [20], respectively. To combine the multiple representations, we introduce the neural mixture of representations (NMoR), a neural network inspired by mixture of experts [13] that produces fine weights for different item-based representations. It is optimized over the editor-based representation to rank items to editors. The proposed WikidataRec demonstrates a large capacity of leveraging items' contents, relations, and edit history between editors and items into the recommender.
To facilitate further research in this space, we also create two benchmark datasets, a general-purpose one with 220, 000 editors responsible for 14 million interactions with 4 million items, and a more focused one with contributions of more than 8, 000 active editors with more than 200 edits each. We evaluate the system on these datasets against several baselines and analyse the impact of different features and levels of data sparsity on performance. The results are promising, though challenges remain because of unbalanced participation, sparse interaction data and little explicit information about editors' interests and feedback.

Wikidata Data Model
Wikidata consists of structured records stored in form of entities, where each entity is allocated a separate page and described by a set of terms and statements [40]. There are two types of entities, items and properties, situated in different namespaces. The namespaces are identified by URIs using numbers and letters, with the letter relating to items and the letter to properties. Although there are other namespaces in Wikidata, we focus only on these two, as they are most relevant for our recommender system. Source: [42] Items and property pages have a similar interface, starting with labels, descriptions and aliases, which are language-specific, and then moving on to statements, which are language-agnostic. Each statement contains a set of triplet edges (head, relation, tail), capturing the different types of relations between entities in the world, e.g., London − −−−− → capital of United Kingdom. Pages also include sitelinks connecting to Wikipedia articles or other Wikimedia projects [24]. Each page is indexed by a unique identifier. Figure 1 shows an example of a Wikidata item page.

Editors and Editing in Wikidata
Anyone can edit Wikidata. The literature tends to distinguish between two types of contributions: manual ones, carried out by registered or anonymous editors, and bots. Bots are used for repetitive editing tasks, but do not contribute to discussions, modeling decisions, or rules for content creation, and management [24]. Human editors may get involved in any type of activities, though some require specific access rights or ontology engineering skills [28,29]. Furthermore, there are two types of editing tasks which are higher-risk tasks and lower-risk ones. Edits are higher-risk if they affect a larger share of the graph, e.g. property editing; hence not all editors may create properties in Wikidata. Lower-risk edits are, for example adding/changing labels, descriptions, etc. Formally, the Wikidata editing process starts when an editor logs into the platform and contributes by inserting data on an item page using the basic editing interface (See Figure 1). Every action performed by an editor is recorded on the so-called revision history page. In our current work, we focus on item edits rather than properties or other types of contributions. Our aim is to establish whether recommendations are technically feasible with the available data. Previous studies into the Wikidata community suggest that item edits are popular with less experienced editors, who are the main audience for personalised recommendations. Properties or other sorts of works are normally dealt by seasoned and active editors [28,29].

Recommending Tasks to Communities and Crowds Online
There are a lot of works proposing the use of personalized recommendations in online communities and peer-production systems [7,9,44]. Recommendations aim to make work more effective and increase retention, by matching open tasks to people's skills and interests [5]. As noted in the introduction, they can also help new members of a community find their way and contribute [9,28]. In Wikipedia, a first recommender system, SuggestBot, was proposed in [5]. It used article titles, links, and co-editing patterns as main features to recommend articles to editors. A more recent recommender system by [23] represents Wikipedia articles using Graph Convolutional Networks. Both works aim to recommend items (Wikipedia articles) to editors based on features of items and editors. There is no explicit feedback that would confirm an editor's interest in an article. Equally, there is very little information about editors beyond what they have edited so far [23]. Our system is similar in spirit to [5] and [23], though our neural approach incorporates additional item-based features and a mix of representations.
In community question-answering (CQA), several papers develop recommender systems to route users to questions they might be interested in answering, hence improving their engagement on the platform and reducing question answering time [7,21,37,38]. [7,37] model recommendation as a classification problem, using different machine learning techniques. They implement a hybrid approach with content and collaborative knowledge to address the sparsity problem. [21,38] apply graph embedding techniques to tackle the same problem. Similarly, we mix item and relational representations with collaborative filtering to solve in a novel application context.
Another related area is online crowdsourcing, where the aim is to allocate tasks published by a requester to an online crowd. [18,33] employ a probabilistic matrix factorization (PMF) to recommend suitable tasks to crowdworkers based on their previous activities, performance, and preferences. They handle the cold start problem by utilizing predefined categories (e.g., sentiment analysis, translation, image labeling) as additional features to improve the recommendation accuracy.

Evaluating Recommender Systems
Evaluation Methodologies. To evaluate the performance of a recommender system, precision@k and recall@k are the most common metrics for top-k recommendations task with implicit data [36]. Precision@k measures the relevance of the recommended items list, whereas recall@k gives insight about how well the recommender is able to recall all the items the editor has rated positively (or interacted with) in the test set [36]. Moreover, there is a set of metrics that cares about where the relevant item appears in the recommended list. For instance, mean average precision(MAP) and mean average recall (MAR) are popular metric for search engines and are applied to the recommendation task [2]. In these metrics, the relevant items are required to be placed as high on the recommendation list as possible [36]. There is also a family of metrics such as catalog coverage and diversity that pays special attention to editor experience [2]. The catalog coverage evaluates whether the recommender algorithm can generate recommendations with a high proportion of items, and the diversity evaluates how diverse the set of proposed recommendations within a single recommended list [2].

Recommender Datasets.
Recommender datasets are available for items such as movies (using MovieLens) or products (using the Amazon dataset) . In other tasks, such benchmarks are not widely available. The systems discussed in Section 2.3 are mostly evaluated on custom-built datasets derived from the platforms' activity logs. This is also the case with our system, which to the best of our knowledge is the first of its kind for Wikidata. To encourage further research in this space, we hence provide two new datasets for the community to reuse, specified in Section 3.

From Wikidata Dumps to Relational Tables
Wikidata dumps are readily available as JSON, RDF and XML . In our work we use the JSON and XML ones from July 1, 2019. The JSON dump contains all Wikidata pages without their edit history, which is available as XML. We parse the JSON data to extract all Wikidata items along with their corresponding identifiers, labels, descriptions, and statements. For the text data (i.e. label and description), we fetch only the English language version and discard data for other languages, because English is the best covered language in Wikidata [15]. Also, we ignore items' metadata about aliases and sitelinks as we do not use them as features in our system. We transform the parsed data into CSV and import it into a PostgreSQL database (as a Wikidata-items-content table); further processed the raw data containing the items' statements (i.e. claims) in similar way (as a Wikidata-items-relations PostgreSQL table). Next, we work on the XML dump to extract editing information for each individual editor. We only focus on edits that are performed on an item's namespace and ignore all other non-item-related actions, as we do not model this information in our system. For each edit, the following information is kept: who carried out the edit, the item being edited, the timestamps of the edit, and the comment indicating the specific action executed by the editor on the item. They are stored in PostgreSQL (as a Wikidata-editors-edits table).

Sampling and Cleaning
We randomly sample 14 million editing activities performed by 221, 353 editors who are human registered (i.e. non-bots). We sort these activities in ascending order based on timestamps. We refer to this as the Wikidata-14M dataset. We also want to test the system on a denser dataset that contains the editors who interacted with a larger number of items in their editing history and the items that received edits by a high number of editors. The Wikidata-14M dataset is filtered by keeping editors who edited at least 200 unique items during their tenure and items that have been edited by at least 5 different editors. We refer to it as the Wikidata-Active-Editors dataset. We then removed some outliers from both datasets -2.14% of editors in Wikidata-14M and 5% in Wikidata-Active-Editors edited a very large number of items in a very short time relative to the size of their contributions. This is considered as the case for institutional accounts that https://grouplens.org/datasets/movielens/ https://jmcauley.ucsd.edu/data/amazon/ https://dumps.wikimedia.org/wikidatawiki/ publish their data via Wikidata [39]. We remove those accounts and their edits from the datasets, as they are not the main target editors for recommendations. We use both datasets to evaluate our system. The first one Wikidata-14M depicts the actual diversity of editing activities in Wikidata with a mix of active and more casual editors. The second one Wikidata-Active-Editors focuses on more active editors and items -we use it to understand the effects of data sparsity on recommender performance.   Table 1 reports some key descriptive statistics of the two datasets, i.e. number of editors, items and interactions. Figure 2a, 2b and 2c characterise the distribution of edits in the two datasets. For Wikidata-14M, Figure 2a shows the number of items edited by each editor -most editors edited only a few items (<1, 000); while fewer than 100 editors (out of 221, 353) edited more than 1, 500 items. Table 2 illustrates this skewed distribution: the median values are much smaller than the mean values, both per editor and per item.

Datasets Description
It is in line with observations from previous studies in Wikidata and other large-scale community-driven knowledge engineering initiatives [16,29,34], which attest that a small core of contributors are responsible for a majority of the work. This also gives us an opportunity to increase retention in the long tail of editors (novice editors mostly) through recommendations. We also examine items that are edited by multiple editors, which is useful in collaborative filtering: Figure 2b shows that most items have been edited only by small number of editors; the highest number of editors per item is 20. This implies the issue of cold-start for new items, we address it by mining item content and relation information for recommendation (see Section 4). We observe similar, though less pronounced effects for Wikidata-Active-Editors (see Figure 2c and Table 2): the interaction data between editors and items are denser, e.g. the mean and median of the number of items per editor is 1, 244 and 900, respectively, showing a less skewed distribution. Notice we do not have the corresponding Figure 2b (Editors vs. Items) for Wikidata-Active-Editors, because all active editors are filtered and selected, making this statistic no longer meaningful.

Problem Statement
Our problem is defined as follows: given a set of editors, a set of items, and an interaction matrix is either 1 or 0 indicating whether editor has edited item . The task is learning to estimate the preference scores of editors to items so as to rank and recommend new items to editors. Solving this problem is not straightforward: as discussed in Section 3, there exists a high number of items but a low number of interactions between items and editors; many items have only a few edits. Standard approaches using collaborative filtering [35] rely primarily on the item-editor interaction data and do not perform well in our scenario. Intuitively, there are two ways to improve it: by including more information from either editors (e.g. their interests or activities) or items (e.g. their content or relations). In this paper we focus on the latter, as for the former, there is very little descriptive information available for Wikidata editors, a known issue also present in Wikipedia [5,23]. Of course one could try to collect additional information about editors from their activities in Wikidata discussions or contributions to other WikiProjects. This however is not the scope of this paper as we decide to steer away from it to avoid the potential ethical implications.

Approach
We introduce a Wikidata recommender system WikidataRec which is a hybrid model combining item content and relation information with collaborative filtering [17] by means of mixture of experts (MoE) [13] (Figure 3). The matrix factorization (MF) decomposes the item-editor interaction matrix into editor-centric and item-centric edit representations, denoted by and for editor and item respectively. Item content representation, denoted by , is obtained by the sentence embedding model ELMo [26]; and item relation representation, denoted by , is obtained by the graph embedding model TransR [20] building upon the Wikidata knowledge graph. To combine multiple item-based representations, we introduce a neural mixture of representations (NMoR) inspired by MoE, in which we utilize a soft-gating to assign different weights to each representation. More specifically, the network takes four inputs , , and , where , and are added with weights ( , , ) produced from a soft-gating branch whose input is the concatenation of , and . The merged item representation is fed into an element-wise dot product layer together with to predict the preference score . At the training stage, is optimized with the cross-entropy loss over the ground truth label in ; while at testing, it is used to recommend items to the editors. The whole process can be written as, Notice item and editor representations are used in the network as fixed representations (i.e., they are not updated during the training process). They are learned in advance by MF, ELMo, and TransR, respectively. We decide not to jointly tune these representations for efficiency and simplicity reasons, as it is not lightweight to integrate all the models in a whole. We will work on joint learning in our future work. Below we first specify the generation of , , and and then introduce the neural mixture of representations for , and .

Editor-centric and Item-centric Edit Representations
Editing activities are summarized in the interaction matrix . Inspired by [17], we use matrix factorization to decompose into the editor's latent representation matrix × and the item's latent representation matrix × . This is achieved by approximating the target matrix via the matrix product of two low-rank matrices and : Each row in can be seen as a latent representation of editor in terms of its edits. Similarly, each row of describes an item with respect to its edits by different editors. Thus, the prediction formula from Eqn (2) can also be written as: In order to single out , from , we utilize Bayesian Personalized Ranking (BPR) [32] to optimize over it. If we look at Eqn(1), ( ) is its first term.

Item Content Representation
To generate item content representation , we need an embedding model that can learn both semantic and syntactic representations from text. There are various embedding models that learn item representations from words, sentences, or paragraphs. Among them, word embedding models such as Word2Vec [22], GloVe [25] and FastText [14] are widely used for many recommendation tasks [8], but they have limitations: the order of words is ignored, which leads to the loss of the syntactic and semantic meaning in sentences [41]. We therefore considered two state-of-the-art sentence embedding models instead, specifically Embeddings from Language Models (ELMo) and Sentence-BERT. ELMo [26] employs a bi-directional deep LSTM network which takes an entire sentence as input, assigns representation to each word in the sentence, then takes the average of the vectors of words to output a -dimensional vector for the input sentence. Sentence-BERT [31] relies on a bi-directional Transformer network [31] which learns from fix-sized semantically meaningful sentences. We preferred ELMo to Sentence-BERT to generate because of the high computational costs of the latter, particularly on large corpora [30]. Each item in Wikidata has a very short description which naturally serves as input sentence to ELMo. The implementation details are in Section 5.1.

Item Relational Representation
Item relational reprentations are built upon the Wikidata graph over items. We model item relations as a directed labeled graph, where it has a set of nodes to represent items, a set of edges (unweighted) to represent relations between items, and labels to capture the type of relations (see Section 2.1). Over different graph embedding models [3,20,43], we adopt the TransR [20], a representative approach for heterogeneous graphs, to learn low-dimensional ( in this work) relational embedding of items, . It builds entity and relation embeddings by regarding a relation as a translation from head entity to tail entity [20]. Different from other graph embedding methods [3,43] which normally assume the embedding of entities and relations within the same space, TransR represents entities and relations in distinct spaces and projects entities from entity space to relation space via a projection matrix. For this reason, it is selected to model the heterogeneity of items and their relations in the Wikidata graph. Items with similar relations have similar embedding in TransR. The implementation details are in Sec. 5.1.

Neural Mixture of Representations
Having , , , and ready, we present our neural mixture of representations (NMoR) to combine them for Wikidata recommendation. , , , and are fed into NMoR in parallel as fixed representations, where they are not updated during training but learned in advance (as noted in Section 4.2). Item representations , and are added with weights , and generated by an additional soft-gating branch (see Figure 3). Soft-gating is used to distinguish from hard-gating in MoE. In hard-gating, weights are either 0 or 1 for each expert. Soft-gating can be seen as probabilities combining different experts. The motivation for assigning different weights to different item-based representations is to control their respective contribution to the final prediction (Eqn. 1). These weights are also -dimensional vectors as with , , and such that they are tailored to every dimension of feature representations. The soft-gating branch takes the concatenation of , and of size 1 × Z × 3 (corresponding to rows, columns, and channels of the tensor) as input; three 1D convolutional layers (kernel size 1 x 1, filters 1024) followed by ReLu are applied to process the tensor along its channels; the number of their output channel remains 3. The output of the last convolutional layer is thus of size 1 × Z × 3, where each channel corresponds to the weight for certain item-based representation ( , or ). At every column of the tensor, it is softmaxed over channels such that the three values at each column are summed to 1. The merged item representation · + · + · is passed through the element-wise dot product layer with , their similarity value is hence computed. To optimize , we adopt the binary cross-entropy loss, where is transformed into a probability using the sigmoid activation function (·); is the ground truth label of either 0 or 1 in indicating whether interaction between the item and editor has been observed (i.e. positive instances) or not (i.e. negative instances). Since is sparse, we randomly select negative instances to make its number the same to that of positive instances.
indicates how likely item is relevant to editor , we can use it to recommend items to editors at testing.
Traditional weight tuning approach like line search is only suitable for optimizing a limited number of weights, for instance, one parameter for each item-based representation. The proposed NMoR optimizes weights in a neural network which enables a refined weight steering on every dimension of the item-based representation.

Experimental Setup and Implementation Details
We run experiments on the two datasets Wikidata-14M and Wikidata-Active-Editors introduced in Section 3. Both datasets are structured in an editor-item-edits format. For each dataset, similar to [11,12], we follow the hold-out strategy to select 80% items associated with each editor to constitute the training set and use the remaining 20% as the test set. Within the training set, we set aside 10% as validation data for hyperparameter tuning. In Wikidata-14M dataset, 25% of the editors edited only between 2 to 6 different items during their editing tenure on the selected dates; these are cold-start editors. Therefore, we exclude these editors from the training set and include them only in the test set. We do this because the editing data for this type of editor are rather too sparse to be informative during training. However, we use this data in the test set, as we want to evaluate the performance of WikidataRec on cold-start editors. for feature representations is set to 1024. NMoR: To learn NMoR, the batch size is set to 32. The model is learned with Adam optimizer with a learning rate of 0.001 for 100 epochs. ELMo: To generate the item content representations, we start from the Wikidata-itemscontent table in our datasets, and then follow the following steps using Python's Spacy: 1) tokenize and normalize the content (label and description) of each item; 2) remove stopwords, punctuation and single-letter words; 3) use part-of-speech tagging and retained only nouns and adjectives. We pass the resulting corpus as input to the ELMo model. ELMo is pre-trained on the 1 Billion Word Benchmark that contains about 800 tokens of general-purpose data from WMT 2011 [4]. This is very comprehensive such that fine-tuning on Wikidata is no longer necessary. Following [10], we simply forward each Wikidata item descriptor to ELMo to obtain the item-content representation. Notice ELMo produces multi-scale output which we average them. TransR: To generate the item relational representations, we first build the labeled directed graph ( , , ) from the Wikidata-items-relations (Sec. 3.1), where signifies Wikidata items (the head of the triplet), inter-item relations (the tail of the triplet), and relations' types (the relation of the triplet). We use the graph as input to train the TransR model. The original triplets in the graph are the positive instances. We sample a few of them to replace either their heads or tails with wrong components to create negative triplets. TransR is trained with both positive and negative triplets using binary cross-entropy loss. It is optimized with the SGD optimizer for 10 epochs; the batch size is 128 and learning rate is 0.01.

Evaluation Protocols
For evaluation, in order to simulate the practical recommendation scenario in Wikidata, we follow the spirit in [23] to carry out the test: for each editor and the item-editor interaction matrix in the test set, we 1) split the items of editor into half-half where the editor-centric edit representation along with the representations of item-centric edit, content, and relations , and are obtained by excluding the second-half items in in our model; 2) exclude the editor from to generate the representations of itemcentric edits, item-content and item relations for the second half of items in our model; 3) randomly select 200 negative items that were not edited by editor , where we can obtain their corresponding sub-matrix of and obtain their item-based representations; 4) feed ( , , and ) obtained from step 1 to our NMoR along with the item-based representations obtained from steps 2 and 3 to obtain their ranking scores; The evaluation criteria is to measure how well WikidataRec ranks the correct items against the negative items for a given editor. Therefore, we employ Precision@k, Re-call@k, and MAR@k (see Section 2.4). We are also interested in the diversity of the recommended items, as editing a different set of items is a noted behavior in Wikidata community [27,34]. Having a diverse recommendation would allow the editors to discover a wide range of items for the editing.

Results on Wikidata-14M
Comparison to state of the art. We compare WikidataRec against the following methods on Wikidata-14M: BPR-MF [32] is a representative collaborative filtering model that uses matrix factorization (MF) as the underlying predictor and is optimized with a pairwise ranking loss. It is suitable for recommendation scenarios with no explicit editor feedback and personalised ranked recommendation results. eALS [12] is also a MFbased method that is optimized with square loss. It treats all unobserved interactions as negative instances and weights them non-uniformly by the item's popularity. GMF [11] is a neural network-based collaborative filtering which implements MF with cross-entropy loss. It embeds each item and editor in the network and computes their element-wise dot product to predict the relevance score. YouTube-DNN [6] is a neural recommender for YouTube videos, using deep candidate generation and ranking networks. It uses a hybrid of users' activities and content information of users and items and directly learns their low dimensional representations. In this paper, we adapt YouTube-DNN with our item content and relational representations. Table 3 shows the results: WikidataRec achieves the best performance over all with both accurate and diverse recommendations. First, it outperforms the collaborativefiltering (CF) methods GMF, eALS and BPR-MF by a large margin. CF methods only rely on the item-editor interactions data without taking into account the information of items themselves, whereas adding additional item content and relation information significantly improves the recommendation performance in our model (see Table 4). Our model also beats Youtube DNN. Youtube-DNN learns item's content and relational embedding in a neural network with random initialization while we employ additional state-of-the-art embedding models (i.e., ELMo and TransR) to generate them separately which yields superior performance. Ablation Study. We ablate different components of WikidataRec on the Wikidata-14M to understand the effect of each one. Item contents and relations. To justify the importance of item content and relations in WikidataRec, we start with the original collaborative filtering (CF) technique, BPR-MF, and gradually add item content and relational representations using NMoR. The results are in Table 4 which show that values in every metric are increased by considering item content and relations. Particularly, item content representations provide rich information in terms of words and semantic meanings in items, which contribute more in performance   increase; item relational representations in contrast contribute less. We can note each feature's contributions from the amount of increase when each of them is added. We suggest the reason as these relations are learned from the Wikidata graph, which is unevenly and sparingly connected, as discussed in [29]. Neural mixture of representations. To ablate the proposed NMoR, Table 4 further illustrates the results of WikidataRec with and without using it. For the latter, the itembased representations are added with no weights. WikidataRec with NMoR performs significantly better than WikidataRec without NMoR. Embedding models for item content. We compare the ELMo model with two text embedding models, Word2Vec [22] and FastText [14] to generate item content representations. Word2Vec and FastText are lightweight models which we train them from scratch using our Wikidata. The results are in Table 5: ELMo outperforms Word2Vec and FastText clearly. ELMo employs the deep bi-directional Language Model (biLM), which provides a very rich representation about the word tokens and captures the semantic and syntactic meaning of words. This is not the case in the Word2Vec and FastText who utilize shallow neural networks.

Results on Wikidata-Active-Editors
We ran another ablation study of our model on the second dataset, Wikidata-Active-Editors, introduced in Section 3.3. Wikidata-Active-Editors is a subset of Wikidata-14M, focusing on active editors and frequently edited items. The results are in Table 6, in which we note improved results on WikidataRec by adding item content and relations. In particular, content-based information contributed most towards the performance of the model. On the other hand, the contribution of relational information is less than that of content information. Furthermore, the results show that WikidataRec with NMoR performs better than WikidataRec without NMoR. Dataset sparsity. We further study the sparsity of the dataset in terms of its item-editor interaction data. Wikidata is very sparse in nature: a small number of interactions between  items and editors are observed. Wikidata-Active-Editors is denser than Wikidata-14M, but its sparsity is still 99.90%, which means only 1% of interactions are observed over all the possible connections from every item to every editor in the dataset. To study the influence of the dataset sparsity, we extract three sub-datasets from Wikidata-Active-Editors with different sparsity yet roughly the same size (about 6000 editors and 10,000 items): Wikidata-sparse-1 with 99.81% sparsity, Wikidata-sparse-2 with 99.68% sparsity, and Wikidata-sparse-3 with 99.27%. The editors in each dataset have edited more than 200 different items, and each item has edited by more than 16 different editors. Figure 4 show the Precision@5 and Recall@10 on the three subsets. It shows that when the sparsity decreases (from 99.81% to 99.27%, data gets denser), our model performance increases. This again verifies the conclusion in [1].

Discussion and analysis of results on both datasets
Comparing the results of the two datasets (Table 4 and Table 6), we summarize that: 1) adding item content and relations with NMoR improves the performance on both datasets; 2) WikidataRec performs better on the Wikidata-Active-Editors than on the Wikidata-14M; the former is with denser editing data, so having less sparse data is likely to improve the results. This finding is consistent with the conclusion of [1].

Conclusion and Future Work
We present WikidataRec, a hybrid recommender model that recommends Wikidata items to editors based on their past editing activities. The work is motivated by Wikidata's quest for more (and more engaged) editors to keep up with a knowledge graph of growing size and complexity. As the first work of its kind, our focus is on establishing technical feasibility and providing a benchmark for future recommendation research in Wikidata. We do so with solid system implementation and benchmark datasets. Our model is informed by related research in content-based and collaborative filtering in similar verticals, which cannot rely upon explicit feedback for the recommendation task. It uses state-of-the-art models for representation learning and operates by means of a mixture of experts [13]. We employ ELMo [26] for items' content-based representations and TransR [20] for items' relations-based representations. The results, though far from perfect, are promising and could be considered a baseline for future Wikidata recommender work. Based on our experiments, we make three claims for the recommendation task in Wikidata: 1) Collaborative filtering is not enough to recommend Wikidata items, as editing data is very sparse; however, adding item content and relational representations significantly enhances performance; 2) Not each item representation contributes to the final predictions equally. We show how to optimise the weights with NMoR; 3) While our model works better on the denser datasets extracted from Wikidata-Active-Editors data, showing that there is room for improvement.
We plan to extend the work in several directions. First, we plan to run editor studies, including interviews with more or less experienced editors to learn about their current ways to choose what they work on and perhaps uncover how existing technical affordances and interfaces influence such decisions. Second, we aim to experiment with other recommender models, particularly sequential learning and time-sensitive models that encode the temporal aspect and distinguish between older and more recent editor interests, as well as with ways to elicit more information about the editors in a responsible way. One option here would be to recruit new editors for a study in which they would explicitly consent to us collecting such information or would provide additional information about their interests themselves. Also, since the data of Wikidata can be represented as item-item relation graph and editor-item graph, exploring graph neural networks (GNNs) for a recommendation would be an interesting future direction. The advantages provided by the GNNs would provide great potential to advance our recommendation task. Third, topic-recommendation is another area that might interests the editors in Wikidata more than item recommendations. We plan to investigate this area. In addition, we are going to conducting an editor-centric evaluation would provide a space to examine and compare the feasibility and effectiveness of the two algorithms (topic vs. item recommendations).


Education About Standardization as the Foundation for the Expertise Development of SMEs

Considering that small and medium-sized enterprises (SMEs) are seen as the backbone of the global economy and the ones that (against all odds) stand to gain the most from using and developing standards, the study aims at exploring does expertise in standardization exist in SMEs. To collect data, the semistructured interview via e-mail was used, as it seems to be much more comprehensive than the questionnaire. To analyze data, the Critical Incident Technique (CIT) and the qualitative content analysis method were used as the most suitable methods for analyzing data from subject matter experts. The data collected from a total of 28 experts from 14 countries was analyzed. Finally, five conclusions were determined – (1) expertise resides in all SMEs, (2) ... but not the expertise in standardization, (3) ... so external experts might be needed to support them, (4) education about standardization is crucial for the development of expertise in standardization, (5) ... but only as a support for lifelong learning.

INTRODUCTION
Considering that SMEs are usually described as "the backbone of the global economy" [1] -SMEs (against all odds) stand to gain the most from using and developing standards. Unfortunately, SMEs are neither using nor developing standards as much as they could or should be [2]. These authors explained an attitude of small business owners and managers -"I'm behind my production, my client does not like my delivery, my assistant is ill, I had to mend our machine last night, and my best foreman lost his left finger yesterday. I'm too busy to deal with quality, health, or whatever? Oh, environmental issues!" [2].
Compared to large enterprises, SMEs usually do not have a substantial amount of resources to spend on using and developing standardsusually, they do not have the time, competent employees, and money to engage in standards development [2]. Although there may be some reasonable explanation for such associations between financial constraints and costs of either using or developing standards by SMEsanother reason that seems to be far more significant (although mostly underestimated) must be the lack of knowledge, skills, competence, and expertise in standardization ma-tters. Only by acquiring such knowledge, skills, competence, and expertise in standardization matters, smallsized enterprises can compete on an equal footing and achieve sustained success in today's global economy. Contrary to the conventional thinking that formal education about standardization alone shall be enough for acquiring knowledge, skills, competence, and expertise in standardization matterswhen it comes to the small business community, the focus shall be on lifelong learning.
One of the most significant findings reported by the current state of research may be described as the "enormous gap between latent and manifest needs" for education about standardization [3]. This might be explained by the fact that standardization as such "is neither a science nor even a generally accepted academic discipline" [4]. This seems to suggest that standards and standardization still have a long way to be fully embraced by the academic community, espe-cially considering that they cannot be seen as a part of the generally accepted disciplines, such as "engi-neering or social sciences" [5]. Accordinglycontemporary organizations usually do not consider formal TEHNIKA -KVALITET IMS, STANDARDIZACIJA I METROLOGIJA 21 (2021) 4 education about standardization as needed for merely executing some standardization-related tasks [6]. Following the same reasoning, one can attempt to argue if a lifelong career in standardization may be ''an exception and generally not advisable'' [6]. It may be reasonable to assume that such a career may not be "very attractive" to most young engineers [7]. This might be explained by the fact that most stan-dardization experts did not actually start their careers in standardizationthey were usually active in some domain-specific (mostly technical) area, so mostly "they just stumbled upon standardization, fell into it, or were pushed into it" [7]. Тhe global market seldom asks for education about standardization ... and yet again everyone still talks about the same lack of knowledge, skills, competence, and expertise in standardization matters. As stated by John Hill, Standards Manager and the founder of the ICES -"companies and standards bodies need well-educated standardization experts" [3]. Following the same reasoning, Toshiaki Kurokawa, Ph.D. suggested that Japanese companies also face great difficulties to find and keep standardization experts [3]. Similarly, Doede Bakker speaking on behalf of the Dutch Trade Association concluded that organizations that tend to achieve a competitive advantage at the global market may "develop relevant standards themselves" [3]. This might suggest that employees at all levels must be acquainted with the tremendous significance of standards and standardization for their businesses [3]. Given that employees who are currently active in standardization in their organizations may be retiring in the next couple of decadesthe main concern may be if there will be enough successors [3]. As graduates are seen as decision-makers of the future -(formal) education about standardization shall be seen as crucial but only as a foundation for capacity building and expertise development of SMEs.
To date, formal education about standardization has received careful considerations, especially since most academics believe that once standardization is successfully introduced to the university curricula -"general awareness and appreciation of standards' benefits will automatically result" [7]. This may be one of the main reasons why most academics strive for embracing education about standardization at the university level, "preferably as compulsory subject" [7]. The same author suggested that we should not embrace education about standardization at the university level denoting that such graduates would "become very narrowly specialized" and their career development opportunities may be very limited [7]. Instead, the main goal of such formal education shall be (only) to make more people aware of standards and standardization [7]. This seems to suggest that the main goal of such a formal education might be only raising awareness, contrary to the conventional thinking that formal education can do so much more.
To date, a lot has been done to support education about standardizatione.g. World Standards Day (every 14th October), annual workshops organized by the International Cooperation for Education about Standardization (ICES), meetings organized by the European Union Joint Initiative on Standardization (JIS) -Action 3 -Education about Standardization (EaS), the Joint Working Group on Education about Standardization (JWG-EaS), the Standardization and Regulatory Techniques (START-Ed) Group for Education about Standardization, different national workshops organized by national standards bodies (NSBs) aimed at enhancing collaboration between NSBs and academia, global repositories of teaching materials developed by international and European Standards Developing and Setting Organizations (SDOs and SSOs), etc. Although a lot has been donea vast majority of such initiatives are not (yet) broadly embraced by the academic community [4]. The same author suggested that most of the contents are usually onesided, mostly based on marketing, and not adequate for the academic community [4]. This might be explained by the fact that such initiatives usually start from a blank slate and do not consider ''what was already done in that context'' [4]. This may be one of the main reasons why only a few educational establishments at the European level still teach relevant standardizationrelated courses [8].
The number of universities/colleges that offer standardization-related courses at the European level usually lies "somewhere between 10 and 30" and such courses mostly enroll only a few students [3]. This may be explained by the fact that only a limited amount of standardization-related knowledge may be included only as a part of some specific courses as "mechanical engineering, electrical engineering, information technologies, business administration and law") [8]. Unfortunatelyuniversities still do not offer standardization-related knowledge about "technological, economic, and social considerations" [8]. This may be one of the reasons why universities that offer courses on education about standardization may be found in "Asia -China, Japan, and Corea, but also in Thailand, and thanks to the common European-Asian 'Asia-Link' project on education about standardization -Indonesia, Sri Lanka, and Vietnam, as well" [3]. Contrary non-academic courses might be found in almost every countrymainly courses for the use of specific standards [3].
Considering that education about standardization at the university level may be scarce and not enough -"education about standardization should be offered in the form of continuing education ... and as far as the academic community is concernedthis means postgraduate teaching" [6].
Although students may be acquainted with some technical standards during their formal education at universities and colleges -"new standards emerge and existing standards will be changed or withdrawn" so when they get hired by an SME (or large organization) they will need continuing edu-cation about standardization, as well [6]. Considering that ''they are better off when they have had some education in standards as such'' [6] education about standardization shall be seen as crucial but only as a foundation for lifelong learning.

METHODOLOGY
The previous studies by [9] and [10] aimed at exploring perceptions of experts from formal organizations for standardization, professional and industrial associations, consortia, fora, educational establishments, and governmental organizations related to the significance of expertise in standardization for the internationalization of SMEs.
The main findings were derived from five Reserch Questions (RQs), especially considering: (1) use of standards, (2) participation in the standardization process, (3) expertise in standardization, (4) education about standardization, and (5) main sources for acquiring knowledge, skills, competence and expertise in standardization matters.
Although the main findings suggested that expertise in standardization shall be of tremendous significance for the internationalization of SMEs -"the experts' perceptions related to the existence of expertise in standardization in SMEs significantly differ" [10]. This means that -"while some [experts] argued that expertise resides in all SMEs, others strongly suggested that expertise does not exist in SMEs" [10].
Considering that "owners and managers of SMEs must have had some level of expertise that enabled them to start their business activities and successfully carry them forward" [10] the question still remainscan it be expertise in standardization?
To collect dataa semi-structured interview via email was used, following the recommendations by [11]. The data collection was administered in January 2020, as we collected "a total of 28 answers from standardization experts from 14 countries" -"the United States of America, Canada, Australia, China, Japan, Switzerland, Belgium, Netherlands, Spain, Italy, France, Bulgaria, Lithuania, and Serbia" [9].
To analyze data -Critical Incident Technique (CIT) was used, following the recommendations by [12]. Once critical incidents have been determinedwe used the qualitative content analysis method following the guidelines and recommendations by [13,14]. The data collection and the data analysis were previously explained and described in detail by [9].

RESULTS & DISCUSSION
When it comes to the existence of expertise in standardization in micro, small, and medium-sized enterprisesa total of 5 conclusions may be derived -

Expertise resides in all SMEs …
As stated by P10 -''most SMEs have considerable expertise vested in a few owners and employees of that SME; SMEs are most often led by people who [must have] had a high level of expertise that enabled them to start their business activities and successfully carry them forward; the expertise resides in almost every SME; the challenge is to find ways by which that expertise can be used to develop a standard''. This seems to suggest that expertise (or knowledge, skills, and competence as a foundation for such expertise) may be found in most SMEs. These results are consistent with the results of [15] especially considering that owners and managers of small enterprises "have knowledge in key areas of competencies and think they can compete using such knowledge". Contrary, such results are not consistent with the results of [16] who suggested that in SMEs -"there is no place for specialists". These authors suggested that small enterprises "may take advice" from the external world, but such enterprises usually lack "resources and expertise to take such complex technical decisions through a specialist channel" [16]. This may be explained by the fact that owners and managers of SMEs usually "end up making most of the decisions, even the decisions about ban-king, technology, communications, credit facili-ti-es, office equipment, whether to buy a new radio for the factory floor, to the kind of stationery to order"so they do not have time to be concerned with experts [16]. Although one can attempt to argue if Kirzner's theory on entrepreneurial vs. expert knowledge from about 20 years ago may still be relevant in today's global economy (e.g. Kirzner argued that the "entre-preneurs may not have such a domain-specific (expert) knowledge that experts may have, but it is the entrepreneur who recognizes the value and the opportunity of the expert knowledge" ... accordingly, "it is the entrepreneur that hires the expert(s)"), different authors agreed that the absorptive capacity of entrepreneurs shall be seen as crucial -"along with their ability to recognize specialist knowledge" [17].

… but not the expertise in standardization …
As stated by P2 -"experts in standardization are usually the professional staff involved in the standardization process, i.e. technical officers, secretaries or committee managers, etc.; experts and delegates participating in technical committees, are experts in their field of knowledge (e.g. electric equipment for ..., test method on ...) but often not [experts] in standardization, unless they have a leading position like convenors, chairs, etc.)". This seems to suggest that if one can succeed at finding expertise in a small-scale firmit might just not be the expertise in standardization. These results are consistent with the results of [7] especially considering that most experts in standardization are "usually active in a technical area, ... in many casesthey just stumbled upon standardization, fell into it, or were pushed into it". Since, compared to their larger counterparts, SMEs usually do not have a substantial amount of resources to spend on using and developing standardsdeveloping expertise in standardization may not be seen as the current priority. These results are consistent with the results of [18] arguing that owners and managers of SMEs "seem to be too busy dealing with urgent tasks" to give standards and standardization a high priority. These results are also consistent with the results of [19] suggesting that motivation for standardization seems to be low in organizations (large and SMEs) "due to the relatively low attractiveness of the topic".
As stated by P10 -"ideally, an SME would need and thus want to be involved in the development of an international standard; however, it is unrealistic to expect that SMEs have the time, knowledge and expertise by which they could take such a rolе". This seems to suggest that even if an SME wants to get involved (if standardization is made a high priority)there may not be competent employees ready or eager to assume some of the standardization-related roles. Additionally, most small business owners and managers "usually do not know how to delegate" so they "end up making most of the decisions", even the decisions about using and developing standards that are usually "the responsibility of expert professionals in large firms" [20]. These results are consistent with the results of [18] suggesting that large organizations may be expected to deal with standardization matters as "some of the responsibilities and authorities can often be distributed to the lower-level management".
As mentioned by P27 -"participation in the standardization process brings many advantages, but SMEs usually don't have enough knowledge about the standards development process; participation can be an overwhelming and frustrating process for SMEs due to the lack of knowledge and skills in this field". This seems to suggest that even if an SME recognizes standardization as a key strategic activity and decides to get involvedit may still face severe limitations to take standardization-related activities seriously, in terms that large organizations might. This may denote the fact that standardization may be seen as "a management burden for most SMEs" [9]especially, due to the same lack of knowledge, skills, competence, or expertise in standardization matters.

… so external experts are needed to support them
As stated by P21 -"since no one in a small firm has a sufficient level of standardization knowledge, external experts are usually needed to support them". This seems to suggest that due to the constant lack of resources and capabilities SMEs face dailythey may have to explore external sources of knowledge, skills, and competence as a foundation for expertise development. These results are consistent with the results of [21,22] considering that due to severe resource constraints, SMEs are usually more apt to seek and acquire knowledge from external sources.
As stated by P15 -"in general, the number of (low threshold) sources [for acquiring knowledge about standards] seems limited; turning to national standards bo-dies (for information or training), ordepending on the sectorsectoral expertise centersseems to be the most accessible option; engaging external consultants could be the next but more expensive step". This may suggest that SMEs may need some kind of continuous support for acquiring knowledge, skills, competence, and expertise in standardization matters. Given severe limitations small firms face daily, such support shall genuinely come from external sources (e.g. organizations for standardization, professional/industrial associations, consortia, fora, educational establish-ments, even governmental organizations shall be seen as crucial for capacity building and expertise development of SMEs. As suggested by P10 -"the main sources of such information [main sources for acquiring knowledge, about standards] should come from the National Standards Body (NSB) but, given the lack of NSB's resources, that information is usually not available; universities should hopefully be able to fill that gap; university professors and lecturers who have the knowledge and expertise can pass that knowledge and expertise to a wide group of young people who can hopefully take that knowledge with them as they begin and advance their careers". This might suggest thatto achieve a competitive advantage in today's global economy, SMEs must take advantage of fairly extensive courses, seminars, and training offered by formal organizations for standardization, professional and industrial associations, consortia, fora, educational establishments, and governmental organizations, especially aimed at capacity building and expertise development of small enterprises.

Education about standardization is crucial for the development of expertise in standardization
As suggested by P15 -"university education [about standardization] should give experts (to be) a head start related to their awareness of the topic, its relevance, basic features, etc., also, it will increase their ability to identify standardization as a relevant area for their own (career) development". This might suggest that education about standardization shall be seen as crucial for young engineers to acquire some knowledge about standards in today's global economy. If engineers/graduates employed by SMEs (or large organizations) have already acquired some knowledge about standards at the university level, it may be easier for them to trace standards that might be relevant, obtain relevant standards, understand relevant standards, implement relevant standards and evaluate the implementation of relevant standards [2]. Accordingly, it may be easier for engineers to trace relevant sta-ndards development projects, to become involved (and remain effectively involved) in the standards develo-pment, to start new activities in the standards develo-pment, and to evaluate their active involvement in the standards development [2].
As mentioned by P17 -"[university] education about standardization gives additional tools to all graduates that find themselves employed by SMEs (or large enterprises) to integrate standards communities faster and with the knowledge needed to fully benefit from the standardization process". These results are consistent with the results of [2] considering that "students should be made aware of standards (standardization) since they are the future SME employees or SME founders". According to [2] -"experience from Asia shows that two elements are crucial for the integration of standardization in the national educational system: (a) a national policy of education about standardization, and (b) long-term availability of staff to support universities and schools by training teachers, developing materials, access standards, terms of exams, etc.". These results are also consistent with the conclusions of the ICES workshop, suggesting that "the discussions highlight the enormous gap between latent and manifest needs for standardization education as the main challenge; countries like Korea, Thailand, and Turkey show that to bridge this gap, a national strategy is needed, as is a cooperation between a country's government, industry, national standards body, academia, and educational institutions" [3]. This may suggest thatthe needs of different stakeholders must be taken into account for developing such a nati-onal strategy. These results are also consistent with the results of [6] considering that students may benefit from acquiring knowledge about standards, especially since "many of them will have to do with standards".
As stated by P2 -"education about standardization at the university level helps to raise awareness among SMEs to use standardization as a tool for business; education is like a seed you need to plant to grow a sound tree, ... there are several free educational sources available and ready to be used by SMEs for acquiring knowledge about standards, ... but it may be difficult for SMEs to use these sources unless they are aware of the use and the impact of the standardization". This might suggest that the university education about standardization may only be seen as one of the tools for raising awareness about standards and standardization in today's global economy, but not the only one. These results seem to be consistent with the results of [7] suggesting that only by raising awareness among young engineers about standards and standardization "we can ensure that knowledge about standardization will spread into our society and will be recognized and valued as structured common sense". Although we must agree that standardization can not be seen "as a goal itself, but as a very useful supporting discipline" [7] one can attempt to argue if formal education about standardization at the university level shall only serve to make more graduates aware of standards and standardization. As stated by [7] -"numerous are actions available for making the masses aware of the importance of standardization". So, does it have to be the main goal of university education, as well? These results are consistent with the results of [4] given that "learning outcomes of education about standardization in higher education may be significantly higher than raising awareness".
As stated by P26 -"standards as a means of capturing and disseminating knowledge, can make a positive contribution to the economy, growth, and prosperity; education about standardization at the university level is directly related to the development of expertise in standardization of SMEs, especially at a time when organizations need more innovation to remain competitive on the global stage". This may suggest that formal education about standardization at the university level shall be seen as crucial, but only as the foundation for capacity building and expertise development of small firms.

… but only as a support for lifelong learning
As suggested by P7 -"education is a good foundation, but it needs to be upgraded quite a bit for this knowledge to be applicable in a particular work environment". This may be explained by the fact that although education about standardization seems crucial -when it comes to the small business communitythe focus shall be on lifelong learning. These results are consistent with the results of [3] considering that "such knowledge may not suffice, as new standards emerge and existing standards are changed or withdrawn, so once people really get involved in developing or applying standards in their professional life they will need continuous education [lifelong learning], e.g. by means of course offered by NSBs". These results are also consistent with the results of [3] since "people who do the standards work (e.g. who participate in standards committees)mostly do so without any education and training", … so they are not aware that continuing education "could enhance their effectiveness considerably".
As stated by P14 -"SMEs would benefit most from a comprehensive textbook; а really good one is not available; current books are one-sided, especially those prepared by standards bodies, or too difficult; 'Standardization for Dummies' is not available". These results are consistent with the results of [3]considering that "a vast majority of books, e-learning environments, digital lectures and internet courses, bibliographies, and teaching cases ... are written in Chinese, Dutch, Japanese, Korean, Turkish, etc. and as such are not easily accessible". This may suggest that only a small amount of standardization-related litera-ture is written in English and is understandable across the globe. These results are consistent with the results of [6] suggesting that even though "the number of textbooks in standardization has risen ... considering the number of people that spend time developing or using standards, it is just a beginning".
As stated by P25 -"some of the major standards development organizations have fairly extensive training programs, so staff from SME's that are going to be involved in standards with these bodies do have options; if the standard is being developed by a smaller standards body they may not have training or education options; as an alternative, universities are beginning to offer continuing education programs which may include modules on standards, but these are still rather small in number currently". This might suggest that formal education about standardization and such con-tinuing education about standardization are still scarce and not enough. There is enough evidence to suggest that ''education about standardization must find its place in formal higher education and lifelong learning, ... to have future professionals who can use their knowledge about standards and standardization for technology transfer, economic growth, global sustai-nability, orin the case of latecomer countriesfor strategies of 'accept and learn' or of (technology) "catch up" [4].

CONCLUSION
Our findings suggested that expertise exists in all SMEs (but not the expertise in standardization), … so external experts might be needed to support them.
This might suggest that SMEs may need some kind of continuous and long-lasting support aimed at capacity building & expertise development of SMEs.
Our results suggested that the gap can be bridged first by the formal education about standardization at the university level and, second, by the continuing education about standardization (lifelong learning). Contrary to the conventional thinking that formal education shall be enough for acquiring knowledge, skills, competence, and expertise in standardizationwhen it comes to the small business community, the focus shall unanimously be on lifelong learning.
Two aspects shall be paid careful considerationsfirst, speaking about acquiring knowledge, skills, and competence in standardization mattersthe question remainswill such knowledge, skills, and competence be enough for expertise development? SMEs may attempt to acquire some basic knowledge about standards (e.g. quality/environmental aspects)but will it be enough for developing the expertise? Second, as an outstanding performance shall be seen as the characteristic of the expertise [23]a que-stion may beare SMEs capable of outstanding per-formance? The same author explained that -"what is generally accepted from employees is a minimum ac-ceptable level of performance"and "what is desired is an outstanding performance" [23]. This seems to su-ggest that to be competitive and achieve sustained success in today's global economyorganizations mu-st tend to develop expertise [23].
Offering continuous support to small enterprises shall not start from the blank slate but shall consider solutions already available to SMEs around the world. Only through the successful collaboration of stakeholders from organizations for standardization, professional and industrial associations, consortia, fora, educational establishments, and governmental organizations, continuous and long-lasting support for small businesses shall be successfully co-created.

ACKNOWLEDGEMENTS
The studies based on the same data collection (and with different content) were previously published by the EURAS 2020 and the EURAS 2021 Proceedings [9,10].