Deleter: Leveraging BERT to Perform Unsupervised Successive Text Compression

Text compression has diverse applications such as Summarization, Reading Comprehension and Text Editing. However, almost all existing approaches require either hand-crafted features, syntactic labels or parallel data. Even for one that achieves this task in an unsupervised setting, its architecture necessitates a task-specific autoencoder. Moreover, these models only generate one compressed sentence for each source input, so that adapting to different style requirements (e.g. length) for the final output usually implies retraining the model from scratch. In this work, we propose a fully unsupervised model, Deleter, that is able to discover an"optimal deletion path"for an arbitrary sentence, where each intermediate sequence along the path is a coherent subsequence of the previous one. This approach relies exclusively on a pretrained bidirectional language model (BERT) to score each candidate deletion based on the average Perplexity of the resulting sentence and performs progressive greedy lookahead search to select the best deletion for each step. We apply Deleter to the task of extractive Sentence Compression, and found that our model is competitive with state-of-the-art supervised models trained on 1.02 million in-domain examples with similar compression ratio. Qualitative analysis, as well as automatic and human evaluations both verify that our model produces high-quality compression.

Introduction
Text compression can be applied to various tasks such as Summarization, Text Editing and even Data Augmentation where the compressed text can be employed as additional training examples. However, almost all existing approaches require either parallel data, hand-crafted rules, or extra syntactic information such as dependency labels or part-of-speech tag features trees (McDonald, 2006;Filippova and Strube, 2008;Zhao et al., 2018). Even for one model that achieves this task in an unsupervised setting, its architecture necessitates a task-specific "sequence-to-sequenceto-sequence" autoencoder (Baziotis et al., 2019). Moreover, these models only generate one compressed sentence for each source input, making adaption to different style requirements difficult.
In this work, we propose a fully unsupervised model, DELETER, that is able to discover an "optimal deletion path" for a sentence, where each intermediate sequence along the path is a coherent subsequence of the previous one. DELETER relies exclusively on a pretrained bidirectional language model, BERT (Devlin et al., 2018), to score each candidate deletion based on the average Perplexity of the resulting sentence and performs progressive greedy look-ahead search to select the best deletion(s) for each step. 1 Table 1 shows a real example of how the DELETER gradually turns "i think america is still a fairly crowded country by the way ." into "america is a country .", where each intermediate sequence between them is a completely grammatical sentence. Interestingly, the model can also be used to delete extra words in a sentence, such as turning "i work work at a company ." into "i work at a company ." As shown in the table, unlike previous pureblack-box approaches, our model not only provides the final compression, but also exposes the full deletion path, making the compression process more comprehensible and controllable, so that it is easy to freeze certain key words from the original sentence or enforce a certain minimum/maximum compression ratio.
We apply DELETER to the task of extractive Sentence Compression, and found that our 1 We will release all our code and model outputs soon. is a country 7.55 Table 1: The optimal deletion path our model finds (without applying a termination condition) for the sentence "i think america is still a fairly crowded country by the way ." model is competitive with state-of-the-art supervised models trained on 1.02 million in-domain examples. Qualitative analysis, as well as automatic and human evaluations, both verify that our model produces high-quality compression.

The DELETER Model
Our DELETER employs progressive lookahead greedy search based on a pretrained BERT, which is used to assign a negative log likelihood for each token in a sentence to derive a score for each intermediate sentence. The algorithm aims to minimize the average score along the path to ensure that each intermediate sentence is grammatical. Task Formulation Given a sentence, there are finite possible tokens we could keep deleting until running out of tokens. In our setting, we do not go back and add tokens to an intermediate sentence.
We thus formulate the task of Sentence Compression as finding an optimal deletion path along a Directed Acyclic Graph. Each node in this graph is either the original sentence itself (i.e., the root node, which has no incoming edges) or a subsequence of it. Each node has outgoing edges pointing to each sentence resulting from a legal deletion of the current node (note that our model allows deletion of multiple tokens in one step). Each edge is also associated with a cost, which is the Average Perplexity Score (AvgPPL) of all tokens in the sentence as assigned by BERT. We consider a path optimal if the average score of all nodes along the path is minimized. Sentence Scoring The DELETER assigns each sentence in the graph an AvgPPL score as shown in the equation below. Lower score indicates more syntactically coherent sentence. {T i } are tokens present in the current sentence and {D i } refer to the collection of all deleted tokens. Note that each log D i is obtained by scoring the corresponding token in the original sentence. This second term is crucial to the algorithm because otherwise the model would prefer to delete rarer tokens, whose unigram probabilities are lower (i.e., of higher negative log likelihood). Progressive Lookahead Greedy Search DELETER allows deleting multiple tokens at a time, but naively probing for deletions consisting of multiple tokens will result in computation workload scaling with O(m k ) , where m is the the number of tokens in the current intermediate sentence and k is the maximum deletion length for each step. We thus adopt a progressive approach to help reduce the search space. Figure 1 shows how a real probing step looks like for the sentence "she sings with me .". For each step, we first probe for deletions consisting of only one token; if no candidate obtains a score below a threshold, we increase the lookahead length to 2, and so forth. A candidate is above this threshold T if: where α (set to 0.04 in our experiments) is a hyperparameter controlling the extent of increase on AvgPPL for each step. The lower it is, the better the compression quality (with the price of being slower). L root is the number of tokens in the original sentence. We include this term because shorter sentences tend to have lower percentage of increase on AvgPPL, in which case we want to lower the threshold. Note that during inference, this threshold also functions as the termination condition in case we are only allowed to select one sentence from the deletion path (e.g. for Sentence Compression).
During probing, we also want to slightly discourage the model from deleting multiple tokens at a time, since we want it to traverse the deletion path as gradually as possible to have more intermediate compressions. We thus further multiply AvgPPL by L β s when probing for the next deletion, where L s is the length of the current sentence and β (set to 0.04 in our experiments) indicates how "gradual" we want the DELETER to proceed.

Experimental Setup DELETER Model Details
We use a pretrained BERT uncased model implemented by Hugging-Face. 2 To score any token in a sentence, we use the special token [mask] to mask out the target token, and then prepend a [CLS] and append a [SEP], which function as [START] and [END] tokens. We use both tokens to help the model evaluate whether the current first or last token can function as a sentence start or end. This is also the main reason that we did not choose other competitive language models such as GPT-2 (Radford et al., 2019): because these models are not bidirectional, they will be much less sensitive to deletions from the very end of a sentence. However, note that since BERT is not intended to be a language model, 3 we mask one token at a time to obtain the negative log probability of each. The maximum lookahead length in our work is set to 3 to facilitate deletion of phrases which contain multiple tokens, such as "by the way" in Table 1.
Datasets We experiment on two datasets. Google sentence compression dataset (Filippova and Altun, 2013) (2015), we used the first 1, 000 sentences of evaluation set as our test set. The Gigaword dataset (Napoles et al., 2012) with 1.02 million examples, where the first 200 are labeled for extractive Sentence Compression by two annotators (Zhao et al., 2018). 5 Automatic and Human Evaluation Following Wang et al. (2017), we use the ground truth compressed sentences to compute F1 scores. Although F1 can be indicative of how well a compression model performs, we note that their could be multiple viable compressions for the same sentence, which single-reference ground-truth cannot cover (Handler et al., 2019). Thus to faithfully evaluate the quality of the compressions generated by our model, we follow Zhao et al. (2018) and conducted human studies on the Gigaword dataset with Amazon Mechanical Turk (MTurk). 6 Specifically, we sample 150 examples from the test set, and put our model output side-by-side with the two compressions created by the two annotators. 7 The three compressions were randomly shuffled to anonymize model identity. We hire annotators who have an approval rate of at least 98% and 10, 000 approved HITs. Following (Zhao et al., 2018), we employed Readability and Informativeness as criteria on a five-point Likert scale.

Results and Analysis
Automatic Evaluation We report F1 scores on both Google and Gigaword dataset. Since each of the 200 sentences from Giga test set has two references (by Annotator 1 and 2, respectively), we report two F1's following Zhao et al. (2018). As shown in Table 2, our model is competitive (esp. for Annotator #2) with state-of-the-art supervised models trained on 1.02 million examples with similar compression ratio.
We also report F1 results on the Google dataset (Table 4). We can see that for this dataset our F1 is still far away from state-of-the-art. We reason that   this is partly because the ground-truth compressions in Google dataset are based on news headlines (later edited by human) rather than compressions generated directly by human beings. 8 Human Evaluation As mentioned in Section 3, we conducted human evaluation on Gigaword. As shown in Table 4, the Readability of our model is close to that of Annotator #1, while there is a larger gap on Informativeness between the two. This is expected because our model is syntaxaware rather than semantic-aware. For example, the DELETER would readily delete negation words (e.g., from "i do not love NLP" to "i do love NLP"), which usually causes abrupt change in meaning. We leave enforcing semantic consistency by leveraging Neural Language Inference datasets (Williams et al., 2017) as future work. It is also interesting to note that there is a larger gap between Annotator #1 and #2 than between Annotator #1 and our model, indicating that the quality of compression is a fairly subject matter.

Related Work
Sentence Compression task has been investigated by various previous work (Jing, 2000;Knight and Marcu, 2000;Clarke and Lapata, 2006;McDonald, 2006;Clarke and Lapata, 2008;Filippova and Strube, 2008;Berg-Kirkpatrick et al., 2011), where the more recent work tend to adopt a neural approach (Filippova and Alfonseca, 2015; Wang   Kamigaito et al., 2018;Cífka et al., 2018). Our model is also neural-based since it leverages a neural language model. Our work differs from previous work in that it does not require any syntactic information such as POS tags or dependency parse tree, which is an advantage especially for low-resource language where training data for tagger or parser is scarce. We note that Baziotis et al. (2019) also built an unsupervised model for abstractive sentence compression. They trained a "sequence-to-sequence-tosequence" autoencoder to first compress the original sentence, and then reconstruct it. Similar to our AvgPPL, Zhao et al. (2018) also employed average Perplexity (though without our length correction terms) as the reward to a policy network trained with reinforcement learning.
Another characteristic of our model is that we obtain a sequence of sentences which are all valid compressions of the original one, while other models usually generate only one compression.

Conclusion
We introduced DELETER which finds an optimal deletion path of a sentence where each node along the path is a grammatical subsequence. We applied this model to two Sentence Compression datasets and found that they are comparable with state-of-the-art supervised models. Note that Sentence Compression has not explored the full power of this model since it only selects one sentence as the output. We plan to apply DELETER to tasks such as Data Augmentation, where the training data of each epoch is dynamically assembled by randomly sampling an intermediate sentence along the deletion path of each sentence. This approach can also be leveraged as a way to generate adversarial examples (Niu and Bansal, 2018) because deleting a few tokens usually preserve semantics of the original sentence.


Examination of a repetitive process control system

In the present work, the application of repetitive filters in the robust process control systems is examined. The functionality of the proposed system and the improved performance, robust performance and filtering properties has been proven.

Introduction.
By increasing the performance requirements of the modern technological process control systems, there is considerable interest in the classes of non-stationary control systems in the conditions of a priori uncertainty and unmodulated dynamics, some of which are the following known classes of systems: robust systems; systems with parametric compensation; systems with interference absorption; systems using fractal algorithms; adaptive systems; systems with reduced sensitivity; systems with variable structure; extreme systems; repetitive systems; intelligent control systems (using neural networks, fuzzy multi-theory, genetic algorithms, chaos theory, etc.). There are differences between them in structural and parametric terms. They are predominantly governed by the class of control process, the type of control and interfering performance, and the controlled objective. In the case where the parametric disturbances in the processes are dominant and there are periodic external disturbances whose nominal parameters are known in advance but which fluctuate operationally around the nominal ones and one of the main requirements for the performance of the system is the performance and the robust performance of the system, application of robust repetitive systems is recommended as an effective. In this context, the aim of the current work is to study the properties and applicability of the repetitive filters in robust systems and systems with robust properties. [1].

Robust systems and systems with robust properties.
In literature are known systems with internal model MM R [1][2][3], systems with interference absorption DAS [1], and gain scheduled control system GSC [1], as well as the methods and the algorithms for their synthesis.  The free parameter method solves the task of analytical synthesis of robust systems MM R (figure 1) with an internal model for process control in the conditions of a priori uncertainty, under the criteria: (1): robust stability and robust performance of the system for a predetermined in the synthesis parameter set  , satisfying the requirements (1.a); local criteria for quality LCQ (1.b)process with minimal integral squared error, and initial conditions for synthesisa priori know * G ,  ,  , where:  The method of the equilibrium partial absorption equation [2] solves the task for analytical synthesis of systems (figure 2), absorbing the influence of wave structured disturbances on the adjustable magnetite, for controlled process under a priori uncertainty, under the criteria (6): minimum Euclidean norm (6.a) of the members of the equilibrium balanced absorption equation on the adjustable magnitude; equality stock sustainability (6.b) (or optimal module); LCQ (6.c) (process with reordering set const   ) etc., and initial conditions for synthesis: a priori known * G ,  ,  , a representative trend of the system error in operating (simulation) conditions. The control u (6) of the systems (figure 3) has two additive constituents: r u -on the basis controller * R , realizing the aim of control of the respective LCQ and d u -to absorb the impact of interference  . The matrix E , F и C (7) reflect the dependence of the vector y from the controlled vector u , the disturbance  and the condition of the system x . The "nominal" system corresponding to the absorption system is a hypothetical parametric, non-intrusive linear system synthesized at the given LCQ (6.c) to the nominal model * G , determined with the "nominal" controller * R (8).
Corresponding to the disturbance absorbing system (figure 2) "the classical" parametric disturbed a.
b. in real operational conditions (or with adequate computer simulation). o y , u are signal external interference aggregated on the assignment channel o y and/or the loading u , and  are aggregated internal additive or multiplicative parametric and/or structural disturbances with a wave structure in process control. Analytical design of the absorber A in the structure (figure 2) of the system is according (9), where the fluent is determined by a known methodology [1], and the optimum gain value A k is determined based on (10) and initial conditions for synthesis: a priori known  , type of regulator, where:

Repetitive control and ML filters.
The repetitive control is effectively applied in systems for controlled technological objects characterized by periodic signal interference [4,5]. In the structure of the repetitive regulator, the internal model in the ML (Memory Loop) [6][7][8][9][10][11]   p M L (figure 4 and figure 6) is used, which consists of a periodic signal generator allowing compensation of periodic disturbances with a predetermined and fixed value of the period p T . The base ML has the ability to "memorize" the frequency of "cut-off" and effectively to counteract it, through its specific structure as a dynamic system. There are dependency bridges for it (14)-(16): Figure 4. Base ML filter Figure 5. Improved ML filter In the literature is well-known and improved [9] ML (figure 5). It differs from the basic one [6][7][8][9][10][11] because it uses a model of delay, but in a different structure. Unlike basic one (17), the improved MLfilter (19) is a sustainable dynamic system.
The values of the controller setting parameters in the controlled technological process systems are subject to the aim and the criteria presented to the system in its synthesis. They are not a function of the respective ML-filter, which is added to the controller. In this sense, the design of the ML-filter in the control systems is autonomous and is not related to the synthesis of the controller. For designing ML-filters with memory is used "rejecting module bandpass filter criterion". The essence of this method is a sequential frequency correction. The design filter criterion is defined by (20).


extfieldsofstudy:
Medicine
Computer Science

text:
Open Practices and Resources for Collaborative Digital Pathology

In this paper, we describe open practices and open resources in the field of digital pathology with a specific focus on approaches that ease collaboration in research and education settings. Our review includes open access journals and open peer review, open-source software (libraries, desktop tools, and web applications), and open access collections. We illustrate applications and discuss current limitations and perspectives.

INTRODUCTION
OA and OPR can be applied to all forms of published research output (academic journal articles, conference papers, book chapters, etc.). OSS relates to a family of computer tool in which source code (its underlying recipe written using a human-readable programming language) is disclosed. This contrasts with the closed (proprietary) paradigm where each entity (research lab or company) can only rely on their own developments and are not able to study other's works. Similarly, OSH involves the development of physical systems (e.g., microscopes) through the use of publicly shared design information. OD is data (textual or non-textual) that anyone can access, use, reuse, or distribute. It can be materialized in the form of free access databases as well as free digital file formats (a published specification to describe how information is encoded in a computer file). OER encompasses freely accessible digital documents (textbooks, presentation slides, samples, etc.) that are useful for teaching and learning. In addition to impact on science, medicine, and education, the open science paradigm leads to the development of novel interactions and business models in the industry such as open collaboration and open innovation. Interestingly, the open philosophy also contributes to overcome vendor lock-in situations where a customer using a proprietary product or service cannot easily transition to another technology due to incompatibilities, inefficient processes, or contract constraints.
In this Mini-Review paper, we discuss open initiatives that might pave the way to more collaborative digital pathology (DP) in the future.

OPEN PRACTICES AND RESOURCES
In this section, we briefly list the open practices and resources we are aware of at the time of submission (in early summer 2019) in the field of DP.

OA Journals and OPR
The content of an OA journal is open to all, with no access fees. The Journal of Pathology Informatics (http://www. jpathinformatics.org/) is an OA peer-reviewed online journal dedicated to pathology informatics. It publishes all types of papers related to DP, e.g., development and empirical evaluation of computational algorithms; studies and use cases in clinical, research, or education settings; book reviews or literature surveys; scientific conference reports; etc. Articles are distributed under the terms of a Creative Commons license, which allows others to remix, tweak, and build upon the work noncommercially, as long as appropriate credit is given and the new creations are licensed under the identical terms. Interestingly, other scientific journals of wider scope in the field of biomedical research, medicine, or pathology follow the OA principle and accept papers in the field of DP. It includes Diagnostic Pathology (https://diagnosticpathology.biomedcentral.com/) that considers research in surgical and clinical pathology and also focuses on the technological aspects of pathology including virtual microscopy. Authors are the copyright holders of their article, and according to the BioMed Central license agreement they grant to any third party, in advance and in perpetuity, the right to use, reproduce, or disseminate their article. Similarly, the multidisciplinary and OA journal Nature Scientific Reports regularly publishes papers in the field of DP (https://www.nature.com/search?q= digital+pathology). Obviously, our Mini-Review paper in the Computational Pathology special issue of the Pathology section of Frontiers of Medicine (https://www.frontiersin.org/journals/ medicine/sections/pathology) suggests that other OA journals will also provide the opportunity to publish papers in the DP field.
OPR is an emerging practice with various alternative review methods that seek to make the peer review process more transparent. Generally, scientific conferences implementing OPR mechanisms are still rather few, and well-known venues for DP [European Conference on Digital Pathology (ECDP), and the Computational Pathology Symposium at the European Conference of Pathology (ECP)] did not follow these principles so far. Notably, the reviews of the latest edition of the MICCAI workshop on Computational Pathology (COMPAY) were organized in a single-blind fashion and were made public along with the final version of the papers (https:// openreview.net/group?id=MICCAI.org/2019/Workshop/ COMPAY). Other conferences with a broader scope, such as MIDL (Medical Imaging with Deep Learning) and ICLR (International Conference on Learning Representations) have also published papers with some DP content after an open review evaluation process.

OSS for Image Analysis and Collaborative Research
An OSS is a computer tool in which the source code (sets of operations written in plain text according to a computer language) is disclosed. It is released under a license in which the copyright holder grants users the rights to study, change, and distribute the software to anyone and for any purpose. A permissive license allows redistribution with minimal requirements (e.g., requiring little more than attributing the original portions of the licensed code to the original developers in future derivative works) while a copyleft license stipulates that the same rights have to be preserved in derivative works created later.
There is a long tradition of OSS in the bioinformatics and biomedical imaging communities. More than 30 years ago, NIH Image and ImageJ have been pioneers as open tools for the analysis of scientific images (1). In the DP field, the first opensource initiatives date back to the early 2000s. In the following, we present OSS according to their types: libraries, algorithms, user-friendly desktop tools for isolated image analysis, and userfriendly web-based tools for collaborative image analysis. At the time of submission, about fifty software repositories tagged with "digital pathology" can be found on the open-source Github repository (https://github.com/search?q=digital+pathology). In the following subsections, we list published software tools that have already had some impact or are promising for the field.

Libraries
Libraries are foundational tools that are not directly usable for end-users (e.g., pathologists or biomedical researchers) but that enable the development (by computer scientists) of a wide variety of downstream software. In particular, it includes tools to extract the imaging data from files generated by digital slide scanners, or to convert these into other image formats readable by existing software. The development of Openslide (https://openslide.org/, permissive license) was initiated at the Carnegie Mellon University. It enables to extract image data from proprietary DP slide scanner formats. Bioformats (https://www. openmicroscopy.org/bio-formats/, copyleft license) is a library developed by the Open Microscopy Environment consortium for reading and writing image data using standardized, open formats. As it was originally designed for low-or mediumspatial resolution microscopy images, in its early days, this library combined with an OMERO server (2) was not effective for processing high-resolution pathology images. Latest versions improve support of DP images and propose a new standardized, pyramidal, open image format (3). The DICOM-izer in Orthanc software [https://www.orthanc-server.com/static.php?page=wsi (4), copyleft license] is a stand-alone, cross-platform commandline tool to convert a whole-slide image (WSI) from a non-DICOM format to DICOM (following Supplement 145). It has to be noted that none of these libraries support all existing DP formats. For example, Bioformats does not explicitly support native slide scanners format from 3DHistech, Philips, and Sakura, while OpenSlide does, but OpenSlide does not support PerkinElmer Vectra format while Bioformats does. Therefore, currently available software (see below) have to combine these libraries to cover the wide range of available formats. Other open-source tools related to image handling have been developed previously (5, 6) but their use has remained rather limited so far as they were not neither maintained nor seamlessly integrated into widely applicable, usable software.

Algorithms and Image Analysis Packages
Image analysis packages are implementations of image analysis algorithms or workflows for object classification (e.g., cell types), region segmentation (e.g., tumor delineation), object counting (e.g., nucleus counting), or image quality control (e.g., artifacts detection). These packages are often based on foundational libraries including the aforementioned OpenSlide, scikit-learn (machine learning), Tensorflow, PyTorch or Keras (deep learning), or OpenCV (computer vision). They are often standalone command-line tools developed by computer scientists for computer scientists. SLDC (https://github.com/waliens/sldc, permissive license) is a framework for object detection and classification in multi-gigapixel images. HistoQC [https://github. com/choosehappy/HistoQC (7), permissive license] is a tool to perform quality control of digitized slides, e.g., delineating artifacts and discovering cohort-level outliers based on a combination of various image metrics. BIRL (https://github. com/Borda/BIRL, permissive license) is a software package for the benchmarking of whole-slide image registration methods. Other collection of scripts for stain/color normalization, focus quality assessment, and deep learning are available on Github as previously mentioned.

Desktop Applications for Image Analysis
Existing desktop software tool (i.e., running on an individual computer) for bioimage analysis such as ImageJ/FIJI [https:// fiji.sc/, (1)], CellProfiler (https://cellprofiler.org/, permissive license), or Icy [http://icy.bioimageanalysis.org/ (8); copyleft license] can be used in DP studies [e.g., (9) used Icy for kidney tissue analysis]. Other specific tools were recently developed with features tailored for DP or to exploit more efficiently large image files. QuPath [https://qupath.github.io/ (10); copyleft license] was initiated at the Queen's University Belfast. It is a crossplatform, user-friendly, WSI viewer that incorporates various features for whole-slide images, in particular tissue microarrays, including dearraying, stain estimation, cell detection and feature computation, trainable cell classification, batch processing, and survival analysis. It can be complemented by image analysis routines written in ImageJ macro language, Matlab, or call machine learning algorithms from WEKA. ASAP is another desktop application (https://computationalpathologygroup. github.io/ASAP/, copyleft license) initiated at the Radboud University Medical Center. It can run on 64-bit Windows and Linux machines. It offers whole-slide image viewing and annotation functionalities, as well as mechanisms to overlay segmentation or likelihood maps produced by machine learning algorithms. Orbit (http://www.orbit.bio/ https:// github.com/mstritt/, copyleft license) was initiated at Actelion Pharmaceuticals Ltd. It implements various algorithms for tissue quantification and it offers interfaces to connect to an OMERO image server.

Web-Based Applications for Collaborative Image Analysis
A web-based application is a program that is accessed over a network connection using the Hypertext Transfer Protocol (HTTP) in a web browser. In such a setting, processing operations (such as image analysis tasks) can be initiated by the end-user via the web interface but they are executed on servers (on the intranet, on the internet, or in the cloud).
Although there are several OSS web-based tools for the visualization of WSI (e.g., caMicroscope, http://camicroscope. org/, permissive license; Microdraw, http://microdraw.pasteur. fr/, copyleft license), to our knowledge, there are only two that also offer advanced features to work collaboratively, such as user and project management, and seamlessly integrated tools for semi-automated image analysis.
The first one, Cytomine [https://www.cytomine.org (11), permissive license], was initiated at the University of Liège in 2010. With Cytomine, multiple remote collaborators can organize their imaging data into projects with secured access. This tool relies on generic data models that enable to semantically delineate regions of interests in images in a standardized way (using ontologies and metadata), e.g., for histology, cytology, and other imaging modalities [e.g., multispectral; (12)] without restriction of application domain. It also provides mechanisms (RESTful API, Python and Java clients, container technologies) to readily proofread and share image quantifications (e.g., cell classification, tumor delineation, or cell counting) produced by any computer vision or machine/deep learning-based algorithms, encapsulated into containers (13). Cytomine can also be installed as a desktop software tool but then without collaboration functionalities. It can also be used through Icy desktop application given the flexibility and openness of these two software tools (14).
The second web tool, called Digital Slide Archive [https:// github.com/DigitalSlideArchive, (15), permissive license], was initiated at the University of Atlanta and builds upon the Cancer Slide Digital Archive (see next section). It relies on similar technologies and includes HistomicsTK, a library providing algorithms for image analysis tasks such as color normalization and deconvolution, cell-nuclei segmentation, and positive pixel counting. It is specialized for cancer investigations; hence, its applications are fewer, so far.

OA Collections for Research or Education Purposes
Until recently, technical challenges due to the large amounts of imaging data generated by slide scanners made it difficult to easily share WSI between remote labs. As a consequence, large data sharing initiatives are still rather few although very promising for both education and research purposes. In the recent years, datasets of significant sizes have been published in the context of image analysis challenges or as companion of research papers. The Camelyon challenge [https://camelyon17.grand-challenge. org/, (16)] used Google Drive to share 1399 H&E-stained sentinel lymph node sections of breast cancer patients for automated detection and classification of breast cancer metastases. Each WSI has slide-level label indicating whether it contains no metastases, macro-metastases, micro-metastases, or isolated tumor cells. A subset of 209 WSIs have detailed hand-drawn contours for all metastases. The PanNuke dataset [https://jgamper.github.io/ PanNukeDataset/, (17)] contains 216.4K labeled nuclei from more than 20K WSI at different magnifications. The ANHIR challenge (https://anhir.grand-challenge.org/) for Automatic Non-rigid Histological Image Registration presents hundreds of different types of histopathology tissue (lesions, lung lobes, mammary gland) stained with different dyes and where landmarks have been manually annotated to assess image registration performances.
Other data collections are accessible more easily for endusers through web viewing applications. The Cancer Slide Digital Archive (https://cancer.digitalslidearchive.net/) hosts tens of thousands of WSIs from The Cancer Genome Atlas. The Image Data Repository [(18), https://idr.openmicroscopy. org/tissue/] currently hosts two DP imaging collections. The Cytomine Open Data Collection (https://cytomine.coop/ collection) currently offers tens of high-quality WSI of animal and human histology sections originally used in education settings (http://www.histology.be), with an associated license for each image. The University of Leeds Virtual Pathology Project (https://www.virtualpathology.leeds.ac.uk/) hosts more than 385K digital slides, but these are the property of the University of Leeds and software used to visualize them are not OSS.

DISCUSSION
Without concrete applications, open initiatives would have no practical value in DP. It is remarkable to note that aforementioned OSS and OD collections were successfully used in DP in the recent years to address tens of diverse biomedical research questions or to teach histology and pathology to thousands of biomedical students. QuPath desktop software was used in various fields including oncology and immunology (19), cardiovascular diseases (20), or multiple sclerosis (21) (see other publications at https://github.com/qupath/qupath/wiki/Citing-QuPath). Cytomine web application was used by researchers in various fields including machine/deep learning (22), learning analytics, image analysis in nephrology [e.g., when combined with Icy; (14)], osteoarthritis (23), or lung cancer (24) (see other publications at https://uliege.cytomine.org/#publications). Cytomine was also used in education settings by more than 15,000 users from 85 countries (including developing countries), through a massive open online course (25).
Overall, these tools are being increasingly used in ongoing international research projects worldwide. This broad community of users allows continuous improvement of OSS provided that financial means are set up to guarantee their sustainability beyond the research projects that helped to initiate their development. In order to translate these software tools into production environments (practical pathology courses with thousands of students, or in clinical routine workflows), it is required to provide services (such as training, support, maintenance, specific software developments, etc.) and to rely on pragmatic business development models as these additional efforts could hardly be provided by original authors of these tools. For example, an open, not-for-profit, cooperative company was founded to improve Cytomine software sustainability (http://cytomine.coop). While these kinds of initiatives ease the use of open tools, further efforts should be undertaken to spread their use, particularly in developing countries.
The availability of OD collections also enabled new discoveries and applications; e.g., the Cancer Slide Digital Archive was used to train deep learning models for the classification of adenocarcinoma and squamous cell carcinoma. These models were subsequently applied to independent datasets of FFPE tissues and were also successfully used to directly predict specific mutated genes from pathology images (26). However, there are some limitations with some OD collections in DP. With some of them, there is a lack of precise, semantic, annotations to train deep learning models, and/or a lack of metadata [which can lead to biases if images are reused without precaution; (27)]. In some cases, an associated license that clearly defines the limits of their reuse is missing. Moreover, although these collections are of consistent size, they remain well below what a pathology laboratory deals with routinely, which raises the question of the representativeness of these current collections and derived AI models.
Overall, not all open initiatives have been as successful as we have just presented it. For example, some software packages have hardly been used beyond the laboratory in which they were developed. We believe future community efforts should be undertaken to maximize their reusability. Building a comprehensive, searchable, database of existing tools, such as Bioimage Informatics Search Engine (BISE, http://biii.eu), should be encouraged. In addition, continuous efforts should aim at integrating latest libraries and image analysis packages into collaborative OSS platform presented herein to avoid duplication of efforts. These collaborative software tools might also be further enhanced to disseminate data collections with detailed annotations and metadata and therefore enable the organization of high-impact studies or challenges. Indeed, we believe open and collaborative principles should be further followed to accelerate scientific progress and hence societal impact. In our opinion, individualist practices (where imaging datasets, algorithms, quantification results, and associated knowledge are still often stored and analyzed within the restricted circle of a specific laboratory) should be surpassed. Overall, open initiatives highlighted in this paper might pave the way to more collaborative DP in the future.

AUTHOR CONTRIBUTIONS
The author confirms being the sole contributor of this work and has approved it for publication.


extfieldsofstudy:
Computer Science
Mathematics

text:
Optimal Online Transmission Policy for Energy-Constrained Wireless-Powered Communication Networks

This work considers the design of online transmission policy in a wireless-powered communication system with a given energy budget. The system design objective is to maximize the long-term throughput of the system exploiting the energy storage capability at the wireless-powered node. We formulate the design problem as a constrained Markov decision process (CMDP) problem and obtain the optimal policy of transmit power and time allocation in each fading block via the Lagrangian approach. To investigate the system performance in different scenarios, numerical simulations are conducted with various system parameters. Our simulation results show that the optimal policy significantly outperforms a myopic policy which only maximizes the throughput in the current fading block. Moreover, the optimal allocation of transmit power and time is shown to be insensitive to the change of modulation and coding schemes, which facilitates its practical implementation.

I. INTRODUCTION
Wireless-powered communication networks (WPCNs), which usually consist of a hybrid access point (H-AP) and several user equipments (UEs) [1], have drawn significant attention recently. The system performance in terms of different metrics (e.g., throughput [2], outage [3], energy efficiency [4]) for various scenarios (e.g., point-to-point [5], two-hop relaying [6], multiple-input and multiple-output (MIMO) [7]) have been thoroughly investigated. However, most existing works devoted their efforts to studying the system performance of only one time block (slot), where all the harvested energy is exhausted immediately without exploiting long-term energy storage. In practice, due to the variability of the communication channel quality, it is more reasonable to store part of or even all the harvested energy in the battery when the channel undergoes deep fading. Thus it is of great importance to study the transmission policy for optimizing long-term system performance with long-term energy storage capability.
Some research efforts have been devoted to improving the long-term system performance. Considering two simple online transmission policies for a single-user WPCN, the limiting distribution of the stored energy at the UE as well as the outage performance of the system was investigated in [8].
In [9], the data rate maximization problem of an orthogonal frequency division multiplexing (OFDM)-based WPCN was studied. To jointly optimize the subchannel allocation and the power allocation over time, an offline algorithm and an online algorithm were designed for the case of non-causal channel state information (CSI) and causal CSI, respectively. Considering the variation of the CSI and the evolution of the battery state over slots, the long-term system performance of a two-user WPCN in an infinite horizon was studied in [10]. Based on the theory of Markov decision process, the optimal online policy was obtained to maximize the long-term system throughput. After that, the authors in [11] extended this work to a full-duplex scenario where the H-AP transfers energy and receives information data simultaneously. The corresponding optimal online policy for the full-duplex case was obtained and the long-term performance gap between the full-duplex WPCN and the half-duplex WPCN was also discussed. However, the temporal correlation of the time-varying channels, which can be exploited to improve the system performance, was not considered in these works. Also, the H-AP in these works, e.g., [8]- [11], was assumed to equip with an infinite power supply and hence energy consumption of the system has not been a consideration in the previous studies.
In this paper, we focus on the long-term throughput performance of a WPCN with limited system energy budget. More specifically, considering the H-AP with a finite amount of energy, we design an optimal online transmission policy to maximize the throughput over an infinite horizon. The contribution of the work lies in both the modeling and solution development of the throughput maximization problem. First, during problem formulation, the finite state Markov channel (FSMC) model is adopted to capture the temporal-correlation behavior of the fading channel. Moreover, practical aspects including circuit power consumption and efficiency of the power amplifier are considered to evaluate the total system energy consumption. Then, we formulate the problem as a constrained Markov decision process (CMDP) problem and solve it optimally via the Lagrangian approach, where a bisection search is introduced to update the corresponding Lagrange multiplier. Subsequently, the long-term system performance under various scenario is studied via numerical simulations. In particular, the impact of the system parameters on the system performance is thoroughly discussed, which provides practical insights on the design and implementation of the WPCN.

II. SYSTEM MODEL
As shown in Fig. 1, we consider a WPCN consisting of a H-AP and a single-antenna UE in this paper. The H-AP is equipped with a directional antenna and the UE is driven by a rechargeable battery with maximum capacity B max . A time-correlated block fading channel is considered between the H-AP and the UE, where the channel power gain remains constant in a block but varies from one to another. In block t ∈ 1, 2, · · ·, the channel power gain is expressed as where θ t is a random variable capturing the multipath fading, d is the distance between the H-AP and the UE, and α is the path loss exponent. In each block, there is a wireless energy transfer (WET) period and a wireless information transfer (WIT) period. The UE first harvests energy from the H-AP and stores it in the battery during WET, and then transmits its data to the H-AP utilizing the energy stored in the battery during the following WIT. In this paper, we aim at maximizing the system throughput over an infinite horizon under a given energy budget constraint. This considered problem can be formulated in the framework of a CMDP which consists five elements: the system state space S, the action space A, the probability transition matrix P, the reward function r(·), and the cost function e(·). In the following, detail descriptions of these five elements are provided.

A. System States
For the considered system, the optimal policy is constructed at the H-AP based on the channel information and the battery information. We assume that in the current block, perfect CSI as well as the UE's battery information is available at the H-AP (In practice, this information can be acquired in the training phase at the beginning of each block). Correspondingly, in block t, the system state s t ∈ S consists the channel state h t ∈ H and the battery state b t ∈ B, i.e., s t = [h t , b t ]. Similar to the works in [12]- [15], quantized system state is considered in this paper. Specifically, the system state space S is expressed as S = H × B, where H {1, 2, ..., K} and B {0, ..., l, ..., L} define the set of channel state and battery state, respectively. The battery is at state 0 when the stored energy is exhausted.
In practice, the channel in a communication system is generally time-correlated. As stated in [15]- [17], the timevarying behavior of the fading channel can be well captured by the FSMC model. Accordingly, in this paper, we separate the channel gain by a set of boundaries, i.e., Γ = {Θ 1 , Θ 2 , ..., Θ k , ...Θ K+1 } × d −α , where Θ k varies in an increasing order with Θ 1 = 0 and Θ K+1 = ∞. In the t-th block, the channel state h t ∈ H is said to be at state k (i.e., We assume that there is only an one-step channel state transition from block to block. Denoting π k as the steady state probability of the channel being at state k. With equiprobable partition of the channel gain (this is a reasonable and commonly adopted technique in a FSMC model, cf. [13]- [15]), i.e., π k = 1 K , ∀k ∈ {1, 2, ..., K}, the fading boundaries Θ k can be obtained by solving the following equations: where ρ(θ t ) is the probability density function of the variable θ t . When channel is at state k, i.e., h t = k, the quantized value of the channel gain is Similarly, the available energy in the battery of the UE is discretized into L quantum. Denote Q as one energy quantum level of the battery, then the maximum capacity of the battery is B max = LQ. In the t-th block, the battery state is said to be at state l (i.e., b t = l) if ⌊ Bt Q ⌋ = l, where B t is the available battery energy at the beginning of block t.

B. Actions, Reward, and Cost Functions
At the beginning of each block, the H-AP makes a decision according to the current system state and reports it to the UE such that the system is well scheduled during the following WET and WIT procedure. The time duration of each block T is divided into two orthogonal time slots: τ E t for WET and τ I t for WIT with τ E t + τ I t ≤ T . Let P E t and P I t be the transmit power of the H-AP for WET and the transmit power of the UE for WIT, respectively. Then, the action adopted in block t (denoted by a t ) contains four elements, i.e., a t = {τ E t , τ I t , P E t , P I t }. For a given system state, different actions come with different rewards and costs. In our work, we consider the throughput per block (defined as the data bits transmitted in one block) as the immediate reward and the energy consumption per block as the immediate cost. Denote the feasible action set at state s t as A(s t ). For a given state s t and an action a t ∈ A(s t ), the immediate reward, i.e., r(s t , a t ) : S × A → R, is defined as where W is the bandwidth of the considered system, σ 2 = N 0 W is the thermal noise power (where N 0 is the noise power density), and the factor ζ characterizes the discrepancy between the achievable rate and the channel capacity due to the use of practical modulation and coding schemes [4]. The corresponding immediate cost, i.e., e(s t , a t ) : S × A → R, is expressed as where the first two terms capture the energy consumption at the H-AP and the last two terms describe the battery consumption at the UE. Specifically, 0 < ϑ AP < 1 is the power amplifier efficiency of H-AP. Hence the first term in (4) presents the energy consumption of the power amplifier during WET. P CAP is the circuit power at the H-AP. Hence the second term in (4) accounts for the energy consumption of the circuit during WET. For the battery consumption at the UE, stands for the energy consumption of the UE during WIT, where ϑ U and P CU denote the power amplifier efficiency and the circuit power at the UE, respectively. Finally, is the energy accumulated in the battery in block t, where η is the energy conversion efficiency and G A is the antenna gain at the H-AP during WET. Obviously, the value of e IT t − e AC t can be either positive (battery consumption) or negative (battery accumulation).
By the conservation of energy, both r(s t , a t ) and e(s t , a t ) are nonnegative. Since the available energy of the UE in block t is limited by the current stored energy in the battery, the feasible action set at system state s t can be given as: where P E max is the maximum transmit power of the H-AP.

C. Transition Probabilities
Denote the system state in block t and t + 1 as s t and s t+1 , respectively. For an adopted action a t , the transition probability from state s t to state s t+1 can be given as where (a) holds by definition and (b) holds for the independence of the channel state evolution from the battery state and the action. In the following, we calculate the channel state transition probability P(h t+1 |h t ) and the battery state transition probability P(b t+1 |h t , b t , a t ), respectively.
The channel state transition probability, which is closely related to the time-varying behavior of the channel gain, can be described by the level crossing rate Λ(Θ) [15]- [17], i.e., the average number of times that the instantaneous value of θ t crosses a given level Θ. Specifically, the channel state transition probability from state h t to h t+1 can be approximated by the ratio of Λ(Θ) divided by the average number of blocks the value of θ t falls in the interval associated with the state h t . Similar to [13]- [17], we assume that the channel state transits between its adjacent state only (the validity of this commonlyused assumption has been verified in [16]). Then, the channel transition probabilities can be approximated as On the other hand, the battery state transition can be described as follows. If b t+1 < L, otherwise, where δ(·) is the indicator function.

III. CMDP FORMULATION AND THE OPTIMAL POLICY
In this section, we formulate the CMDP problem and provide the corresponding optimal solution.

A. Problem Formulation
For a system in the long run, a policy µ is a sequence of decision rules, i.e., µ = {µ 1 , µ 2 , ...}, each in which is a function mapping from the system state s to the action to be taken, i.e., µ t : S → A, ∀t. A policy µ is said to be stationary if the decision rule in it is independent with time, i.e., µ 1 = µ 2 = · · · . If a policy is stationary and deterministic, then it is called a pure policy. To model the imperfect operation of the system in Fig. 1, we introduce a factor λ ∈ [0, 1) to capture the probability that the system hardware survives from a operation failure in a block. Correspondingly, as described in [18], for an available stationary policy µ, the long-term throughput of the system can be defined as and the long-term energy cost of the system can be defined as When λ approaches 1, the discounted functions defined in (14) and (15) converge to their expected average values [18], respectively, which are defined in the form of where N is the number of blocks. Thus (14) and (15) can be interpreted as the expected average throughput and the expected average energy cost per block, respectively.
In this paper, we aim at finding an optimal policy µ * such that the long-term throughput is maximized under a given energy budget E th . This policy can be obtained through solving the following CMDP problem:

B. The Optimal Policy
As shown in [18], the CMDP problem in the form of (16) can be efficiently solved via the Lagrangian approach, whereby the CMDP problem is transferred into an equivalent unconstrained MDP problem. Accordingly, by introducing a non-negative Lagrangian multiplier β for problem (16) and the corresponding Bellman's optimality equation is: which can be efficiently solved via the Value Iteration Algorithm (VIA) [19] for any fixed β. Correspondingly, the optimal policy with a given β, i.e., µ β = {µ β (s), ∀s ∈ S}, can be determined by: As described in [18], the optimal policy of a CMDP problem with a single constraint is composed of two pure policies, i.e., µ β − and µ β + , with β − and β + as their associated Lagrangian multipliers, respectively. The policy µ β − yields the highest energy cost E − that satisfies the energy constraint, while the policy µ β + yields the lowest energy cost E + that breaks the energy constraint. Since J β (s) is a monotonically nonincreasing function of β [20], the value of β − and β + can be efficiently obtained via the bisection search method. With a randomized mixture of µ β − and µ β + , the optimal policy of a CMDP problem can be given by: where the mixing weight parameter 0 ≤ q ≤ 1 can be obtained via solving equation E th = qE − + (1 − q)E + . Correspondingly, the procedures for solving problem (16) is described in Algorithm 1. Since the optimal policy consists of two pure policies, both of which are irrelevant to time sequence. In Algorithm 1, we drop the subscript "t" for convenience. Specifically, initializations are performed in line 1, where n and ε β are the iteration sequence and the error bound for updating β, respectively. The initial value of β + is specified in an incremental method, i.e., increasing the initial value of β + until that the corresponding long-term system energy cost exceeds E th . The VIA is conducted to solve the equivalent unconstrained MDP problem in line 4 and the Lagrangian multiplier β is updated via bisection search in lines 5-13. Finally, with the obtained policy µ β − (s) and µ β + (s), the mixing weight q and the optimal policy are obtained in line 17 and line 18, respectively.
For the implementation of VIA, the candidate actions at each state are quantized. Specifically, τ E , τ I , P E , and P I are discretized into levels of V E τ , V I τ , V E P , and V I P , respectively. Since the update of β is independent from the action space and the channel state space, the computational complexity of . Remark 1: In this paper, we obtain the optimal online policy for the CMDP problem (16)  M m=1 e(s m,t , a m,t ), respectively; the system state transition probability matrix can be expressed as P = P 1 ⊗ ...P m ... ⊗ P M , where P m = [P(s m,t+1 |s m,t , a m,t )] is the system state transition probability matrix of the m-th UE and ⊗ is the Kronecker product. Based on this tuple, the CMDP problem for the multiuser case can be constructed and the corresponding optimal online policy can be obtained similarly via Algorithm 1. (16) 1: Set n = 0, β − = 0, β + , β 0 = β − , specify ε β > 0. Set β = β n and n = n + 1.

IV. SIMULATION RESULTS
In this section, numerical simulations are provided for evaluating the long-term throughput performance of the system. For the practicality of RF energy transfer, a Rician fading channel is considered between the H-AP and the UE [22], [23]. Correspondingly, the PDF of θ t is given by where I 0 is the modified Bessel function of the zero-th order, 2̺ 2 and ς 2 are the parameters representing the power of multi-path and line-of-sight, respectively. Moreover, the level crossing rate Λ(Θ b ) is [17] where f D is the maximum Doppler shift of the channel, θ = 2̺ 2 + ξ 2 is the local-mean fading power and κ = ξ 2 2̺ 2 . Accordingly, practical channel parameters setting in [17] is considered in simulations, where the number of channel states is selected as K = 3, f D is set as 1.34 Hz, and the block duration is set as T = 16 ms, respectively. Similar to [10], we focus on the case of small devices and express the battery size as a function of the reference value  Table I. Moreover, to show the superiority of the optimal policy, the myopic policy which maximizes the throughput in only the current block is used as the benchmark. For legibility,   in the simulation results, we mark the optimal policy and the myopic policy as "Optimal" and "Myopic", respectively.
To investigate the impact of the energy budget and the communication distance on the system throughput performance, we first depict the long-term throughput as as a function of the energy budget E th for different value of d. As shown in Fig. 2, the optimal policy outperforms the myopic policy in all the considered cases. The long-term throughput is shown to be increased with E th . This is because that a larger E th means more available energy budget. Due to the limitation of transmit power and the battery capacity, the system performance becomes saturated when E th is exceedingly large (see the case of d=10 m). On the other hand, since the signal attenuations during WIT and WET are decreasing functions of the communication distance. As expected, the long-term throughput is shown to be reduced with d.
The maximum battery capacity B max , which limits the maximum available energy at the UE in each block, is expected to have an impact on the system performance. Hence, in Fig. 3, we investigate the long-term system throughput with varying B max . Here, we set E th = 500B ref and d = 10 m. As shown in the figure, the long-term throughput with the optimal policy increases with B max . In fact, a larger B max means a higher ability to handle the fluctuation of the channel state. As B max grows, the performance gain becomes saturate due to the limitation of E th . However, the myopic policy shows a different trend. With the growth of B max , the corresponding long-term throughput first increases and then decreases. This is due to the fact that the myopic policy operates sequentially from block to block and exhausts the battery's energy as much as possible to maximize the current system throughput, which results in a trade-off on B max . Nevertheless, compared to the myopic policy, considerable improvement can be observed when the optimal policy is adopted. As stated in (3), the factor ζ is used to capture the impact from the practical modulation and coding schemes. In Fig.  4, we depict the long-term throughput as a function of ζ with E th = 500B ref and d = 10 m. As shown in the figure, compared with the myopic policy, a high system performance gain is achieved when the optimal policy is adopted. Moreover, the long-term throughput is shown to be slightly decreased with the increasing ζ. For example, with rising ζ from 1 to 5 (about 7 dB), the long-term throughput performance for the optimal policy drops only about 0.296 dB. On the other hand, the impact of ζ on the optimal policy is investigated in Fig. 5. Here, we take the optimal policy with ζ = 1 (i.e., µ * ζ=1 ) as the reference policy and use a binary indicator C to identify the variation of the optimal policy with ζ. Specifically, denote the optimal policy with ζ ′ as µ * ζ ′ , then C = 1 if µ * ζ ′ is identical to µ * ζ=1 . Otherwise, C = 0. As demonstrated in Fig. 5 Fig. 6. The long-term throughput versus the energy conversion efficiency η with different circuit power P C AP and P C U .
value of C equals to 1 and remains unchanged for different values of ζ, which implies that the optimal policy is irrelevant to the practical implementation of the modulation and coding schemes. Lastly, the impact of the energy conversion efficiency and the circuit power on the system performance is investigated in Fig. 6. Here we set E th = 500B ref and d = 8m. As can be observed, the long-term throughput grows with the increasing of η. This is due to the fact that more available energy can be harvested at the UE with higher energy conversion efficiency. On the other hand, although P CAP dominates the circuit power of the whole system, the system performance is shown to be more sensitive to P CU rather than P CAP . Specifically, with the optimal policy, the long-term throughput achieves a performance gain of 1.2 dB at η = 0.75 when P CU decreases 3 dB (from 10 mW to 5 mW), but is almost unchanged when P CAP drops from 0.5 W to 0.25 W. In practice, this intrigues an prior effort on cutting down the circuit power consumption at the UE rather than at the H-AP.

V. CONCLUSION
In this paper, we studied the problem of designing the optimal online policy in an energy-constrained WPCN to manage the transmit power and time durations for both WET and WIT over time-correlated fading channels. Aiming at maximizing the system long-term throughput with a limited energy budget, we formulate the transmission policy design as a CMDP problem, which was later transformed into an equivalent unconstrained MDP problem and solved via the Lagrangian approach. Numerical results showed that the longterm system performance is closely related to the total energy budget, the battery capacity, the communication distance, the energy conversion efficiency, and the circuit power of the system. For instance, the circuit power consumption at the UE has a stronger impact on the system performance than that at the H-AP. Also, the optimal policy was shown to be independent of the choices of modulation and coding schemes.


Towards Cognitive Obfuscation: Impeding Hardware Reverse Engineering Based on Psychological Insights

In contrast to software reverse engineering, there are hardly any tools available that support hardware reversing. Therefore, the reversing process is conducted by human analysts combining several complex semi-automated steps. However, countermeasures against reversing are evaluated solely against mathematical models. Our research goal is the establishment of cognitive obfuscation based on the exploration of underlying psychological processes. We aim to identify problems which are hard to solve for human analysts and derive novel quantification metrics, thus enabling stronger obfuscation techniques.

INTRODUCTION
Hardware components form the root of trust in virtually any computing system. In order to detect fabrication faults, copyright infringements, counterfeit products, or malicious manipulations hardware reverse engineering is usually the tool-of-choice [2,10,14]. While hardware reverse engineering is a highly complex and universal tool for legitimate purposes, it can also be employed with illegitimate intentions, undermining the integrity of ICs via piracy, subsequent weakening of security functions, or insertion of hardware Trojans [10,19]. In particular, Intellectual Property (IP) piracy has become a major concern for the semiconductor industry which causes losses in the range of several billion dollars [14]. Governments and armed forces classify hardware reverse engineering as a major security threat, since malicious manipulations of hardware components in mission-critical systems can even have lethal consequences [10].
Due to the serious threats posed by attacks based on hardware reverse engineering, strong countermeasures are indispensable. The security of most existing obfuscation techniques is assessed exclusively based on technical measures. However, the process of hardware reverse engineering cannot be fully automated yet and the lack of holistic tools forces human analysts to combine several semi-automated steps [23]. Accordingly, cognitive processes and strategies applied by humans in the context of hardware reverse engineering must be taken into account for the development of sound countermeasures. In this work, we describe a novel approach of developing cognitive obfuscation methods considering both technical and human factors during the hardware reverse engineering processes.
Goal and Contribution. Our work is motivated by the lack of metrics to precisely capture the strength of defenses against hardware reverse engineering. Since technical and human factors are inseparably connected in hardware reverse engineering processes, the thorough understanding of the human factor is essential to develop cognitive obfuscation methods. To the best of our knowledge we are the first to define cognitive obfuscation as the approach of impeding hardware reverse engineering based on the analysis of psychological processes. In summary our contributions are: • Exploration of Human Factors. We define our psychological research design to analyze the underlying cognitive processes of hardware reverse engineering in detail. Additionally, we emphasize how reverse engineering can be formulated as a psychological concept of problem solving and identify which insights will perspectively help to develop sound cognitive obfuscation methods. • Improved Hardware Reverse Engineering Tool HAL. We significantly advance the development of our hardware reverse engineering software HAL [12] to adequately capture human behavior as a foundation for this research. • Cognitive Obfuscation Methods. We derive the necessity for novel obfuscation methods based on psychological insights from the inseparable connection of technical and human factors in hardware reverse engineering.

TECHNICAL BACKGROUND
In the following section we define the term hardware reverse engineering and specify gate-level netlist reverse engineering as our main research topic. Additionally, we provide a brief overview on existing obfuscation methods impeding gate-level netlist reverse engineering and emphasize the significance of our novel approach leading to cognitive obfuscation methods.

Hardware Reverse Engineering
Reverse engineering is commonly described as the process of retrieving knowledge or information from anything manmade in order to understand its inner structure and workings [20]. In the context of computer security, reverse engineering is often associated with the analysis of binary programs [29] or hardware designs [8].
Reverse engineering in the field of hardware security is conducted for legitimate as well as illegitimate purposes [10]. On the one hand, security engineers are often forced to conduct reverse engineering to identify counterfeited hardware, security vulnerabilities, or malicious manipulations like hardware Trojans [25]. On the other hand, hardware reverse engineering is also associated with illegitimate actions such as the analysis of hardware designs on various levels to understand their functionality in order to commit IP infringement, weaken security functions, or to inject Trojans [10].
A valuable reference point and target of hardware reverse engineers are Finite State Machines (FSMs). Building the control logic, FSMs form an integral part of virtually any hardware design. Due to its small circuitry and unique structure -consisting of a feedback path, which connects the state transition logic and state memory (see Figure 1) -the FSM can be easily identified in the sea of gates [11].

State Transition Logic
Output Logic State Memory Input Figure 1: Schematic of an FSM circuit in hardware Another valuable target for reverse engineers are implementations of cryptographic ciphers -like the Advanced Encryption Standard (AES) -which are often implemented in hardware to accelerate encryption and decryption processes. Reverse engineering of such implementations can reveal weaknesses or even hard-coded keys in the worst-case. Furthermore, the acquired knowledge can serve as a starting point to insert a hardware Trojan into the given structure.
Gate-level Netlist Reverse Engineering. A gate-level netlist is a representation of a set of logic gates of a particular gate library together with their interconnections [27]. In this work we will focus on Field Programmable Gate Array (FPGA) gate-level netlist reverse engineering. In FPGA netlists combinational logic is usually implemented with Look-Up-Tables (LUTs) and Multiplexers (MUX), while sequential logic is realized with Flip-Flops (FFs).
During synthesis, which transforms a high-level description of the circuit into the gate-level netlist, important humanreadable information is lost, namely (1) meaningful descriptive labels, (2) boundaries of implemented modules, and (3) module hierarchies. The absence of this information drastically complicates the reverse engineering process and forces human analysts to conduct non-automatable sense-making processes [10].

Obfuscation Methods
Obfuscation itself describes a transformation which obstructs high-level information without changing functionality. The goal of obfuscation is to impede the reverse engineering processes.
Hardware designs can be separated in data path and control path. While there is a lack of data path obfuscation, control path obfuscation has received manifold contributions by the scientific community. Many works, e.g., [5], [7], [1], and [6], introduced techniques which modify the behavior of an FSM. Simplified, dummy states are added and the amount of possible transitions is heavily increased, thus obfuscating the original behavior of the FSM. The proposed security analyses of these obfuscation methods almost solely consider technical or mathematical attacks and often prove security in the respective models. However, Fyrbiak et al. presented a variety of attacks against such techniques [11]. These attacks were successful, because the obfuscated parts of the FSMs could be semi-automatically distinguished from original parts. For these distinguishers, human sense-making processes and goal-oriented strategies were indispensable to differentiate both parts and eventually led to practical attacks against presumably secure techniques.
We emphasize, that security assessments and proofs of the aforementioned techniques were solely based on technical models. Incorporating knowledge about the underlying psychological processes faced by human analysts during the reverse engineering of hardware designs might have directly uncovered the anchor points used by Fyrbiak et al. in [11].
An holistic overview of recent metrics and obfuscation methods for hardware can be found in [22].

Need for Cognitive Obfuscation Methods to Impede Hardware Reverse Engineering
Naturally, the absence of high-level information in gate-level netlists requires human analysts to employ sense-making processes during reverse engineering. Hence, human strategies and cognitive abilities influence the success of the reverse engineering process. If we want to develop stronger obfuscation methods it is essential to understand how human analysts are reversing a hardware design. By acquiring a deeper understanding of used strategies or hard problems for a reverse engineer we can derive new metrics to assess the strength of an obfuscation scheme. Eventually, these metrics can lead to stronger approaches for hardware obfuscation based on cognitive insights.

PSYCHOLOGICAL BACKGROUND
Human factors have not been taken into account for the development of obfuscation methods, yet. This is surprising, since successful reverse engineering of netlists requires human analysts to employ their cognitive abilities to design strategies manually. Hence, human factors and underlying psychological processes play a crucial role in hardware reverse engineering and should be included in effective obfuscation methods. To the best of our knowledge we are one of the first to explore relevant human factors and underlying psychological processes for the development of cognitive obfuscation methods. In this section we provide the psychological background required to describe and analyze human behavior in hardware reverse engineering. Our research is based on the theory of reverse engineering and its application to Boolean systems by Lee et al. [16] which will be presented in the following.

Reverse Engineering -A Special Kind of Problem Solving?
Lee and Johnson-Laird [16] explored underlying psychological processes of reverse engineering and defined it as a special kind of problem solving. Problem solving comprises psychological competencies which involve goal-directed thinking and actions in situations, in which the problem solver does not operate routinely [13]. Accordingly, a problem exists when a person does not have the relevant knowledge to produce an immediate solution.
In psychological research, two kinds of problem solving are commonly distinguished [9], [17]. First in analytical problems all relevant information to successfully solve the problem are included. A common example of an analytical problem is the Tower of Hanoi [15]. The challenge of analytical problems are the analysis and sequential utilization of given information [13]. The second type of problems are defined as complex problems which do not specify information to solve the problem. In complex problem situation the problem solver is asked to manipulate information to solve the problem. Complex problems combine different characteristics like non-transparency of information, dependencies between components of the problem, dynamics, or multiple goals [13]. Lee & Johnson-Laird [16] argue that reverse engineering involves both attributes of analytical and complex problem solving. Hence, problem solver in reverse engineering divide a given system (like a netlist) in subsystems which is a common strategy in analytical problem solving. Additionally, dependencies between single components influence the problem solving process in reverse engineering [16]. Dependencies and their influence on the problem solving process are typically allocated to complex problem solving processes [16]. By combining characteristics of both analytical and complex problem solving, reverse engineering can be defined as a special kind of problem solving. In the context of the theory of reverse engineering, Lee and Johnson-Laird [16] conducted five experiments consisting of high-level reverse engineering tasks (see Figure 2 and Figure 3). Participants were asked to draw the underlying circuit which represents the light as on. Both congruent (AND, OR, XOR) and incongruent (NAND, NOR, XNOR) were depicted in the single tasks. Incongruent tasks were harder to solve than congruent tasks. Additionally, the difficulty of reverse engineering was significantly influenced by the number of dependencies between single components which influence the output of a system. Summarized, these results can be seen as a starting point for further research. Nevertheless, it is essential to transfer the experimental setup in the field of hardware reverse engineering to produce higher validity of the results. By the exploration of human processes in realistic hardware reverse engineering tasks, we aim to find more specific answers about how human analysts are reversing a gate-level netlist and which parameters influence the difficulty of solving hardware reverse engineering tasks in realistic scenarios. Knowledge about parameters which force engineers to make mistakes while reversing a chip can be the foundation for the development of cognitive obfuscation methods. In the following we present our novel approach of the exploration of underlying human processes in hardware reverse engineering and how we plan to use insights about human factors to derive novel cognitive obfuscation methods to impede hardware reverse engineering.

A Novel Approach: Cognitive Obfuscation
To the best of our knowledge we are one of the first to derive novel obfuscation methods impeding hardware reverse engineering based on psychological insights. Therefore, we aim to achieve a deeper understanding about underlying psychological processes and how these processes are influenced by human factors. For developing novel obfuscation methods, we first have to explore human processes and to identify the difficulties human analysts are facing during the process of reversing a netlist. In the next step, we can identify and explain common difficulties and recurring mistakes of human analysts with different levels of expertise. We can use these insights to develop cognitive obfuscation methods which force engineers to make these kind of mistakes. Furthermore, we are able to develop cognitive obfuscation methods based on prior results about difficulties and learning processes to test them with human analysts. This enables us to quantify different approaches of cognitive obfuscation and to derive metrics about the strength of each individual approach.
Due to the lack of prior research findings about human processes in realistic hardware reverse engineering scenarios, we formulated the following research questions. They seek to answer if engineers are reversing hardware more efficiently with growing experience, what sort of difficulties occur over time, if these difficulties can be avoided over time, and what other cognitive factors influence the problem solving processes of hardware reverse engineering.

METHOD
In this section we present our research methods, materials, and designs. To answer the research questions above, we employ both longitudinal and cross-sectional research methods for analyzing the behavior of human analysts on different levels of expertise.

Psychological Research Methods
Longitudinal studies are common psychological research methods and often conducted to repeatedly observe behavioral changes of individual participants over a period of time [21]. In comparison to cross-sectional studies, longitudinal studies analyze behavioral changes within one person. The analysis of longitudinal case studies will focus on individual strategies in learning processes and in solving difficulties while working on hardware reverse engineering tasks.
Cross-sectional studies are also commonly used in psychological research and aim to compare behavior between different study participants [4]. Cross-sectional studies will focus on the comparison of used strategies, difficulties between different human analysts, and influences of cognitive factors as well as different levels of expertise.
By combining these two research methods we intend to receive a holistic understanding of underlying psychological processes of humans conducting hardware reverse engineering. Therefore, we analyze behavioral changes in individual human analysts over a period of time on the one hand and compare behavior between different analysts on the other hand. In the first step, we conduct two longitudinal studies by analyzing case studies of individual students. In the next section we outline the study procedures and describe the participants of the two longitudinal studies.

Longitudinal Studies: Participants & Procedures
Both studies are integrated in a novel educational hardware reverse engineering lab course held in winter term 2018/2019 at Ruhr-Universität Bochum (Germany) and at the University of Massachusetts Amherst (USA) [28]. The participants were novice IT security students with backgrounds in computer security, electrical engineering, or computer science. This lab course consists of two parts: the lecture phase and the practical phase. During the lecture phase students acquire relevant knowledge in Boolean algebra, general knowledge about hardware -especially FPGAs and ASICs -Hardware Description Languages (HDLs) like Verilog and VHDL, attack strategies, and obfuscation techniques. In the second phase, students can apply their newly acquired theoretical knowledge by working individually on four realistic hardware reverse Figure 4: The completely revised GUI of HAL engineering projects. The study is included in the practical phase and the following research data will be generated.
Behavioral Data. Behavioral data of reverse engineering processes will be analyzed based on log files. To enable participants to work on hardware reverse engineering tasks, we developed four hardware reverse engineering projects as educational and study material. In all projects (Section 4.3) practical hardware reverse engineering tasks are included which students had to solve by working with our hardware reverse engineering software HAL (Section 4.3). While working with HAL, every single action is recorded automatically and will be saved as a text file including timestamps and actions, e.g. python commands, selecting relevant components in the netlist, or retrieving logic functions. Based on this data, we aim to analyze if students get more efficient in solving hardware reverse engineering tasks, what sort of difficulties or mistakes occurred and how much time they needed to solve them. Please note that the data collection phase has not been finished yet and first results will be presented in more detail at the conference.
Questionnaires. Additionally, participants are asked to answer a number of questionnaires regarding information about socio-demographics, intelligence [26], motivation [24], prior experiences in hardware reverse engineering and related topics, task difficulty [3], and mental effort [18]. Based on this data we aim to analyze which cognitive and motivational factors or levels of prior knowledge influence the success in hardware reverse engineering.
In the following we provide deeper insights about our hardware reverse engineering software HAL and the single projects included in the study.

HAL
Fyrbiak et al. introduced HAL [12], a framework to enable (semi-)automated hardware reverse engineering. The structure of HAL is shown in Figure 5. HAL provides a solid foundation enabling reverse engineers to implement custom plugins for netlist analysis that operate on a graph-based abstraction. This abstraction allows for inspection of arbitrary netlists independent of specific gate-libraries or target architectures.
At the time of publication, HAL was mainly a commandline tool which enabled implementation of arbitrary algorithms through its plugin system. In order to yield verifiable results for psychological analysis we notably improved the GUI of HAL. The design goals were (1) reproducibility, (2) granularity, and (3) more user-friendly. Therefore, we extended the GUI with an event system, a submodule partitioning system, and connected the underlying graph model to a GUI-integrated python context. Figure 4 shows the new GUI of HAL. In multiple dockable widgets a variety of tools are available to the analyst, e.g., a graphical representation of the netlist, detailed information for selected elements, a rich logging window, a python shell and a python code editor.
Event System. The new event system allows for exact tracing of user actions. Regardless of the inducing source (user, plugin, or python code executed live), the core fires detailed events to notify both, GUI and logger, about changes on the . . . Figure 5: Overview of the original HAL architecture from [12] data model. While such an event system is generally used in GUI and game design, we explicitly designed this system to create information which can be used by a psychological analyst to trace the user's actions. The main functionality of the former GUI was to allow for interactive netlist exploration. However, it only displayed an immutable static netlist and was not connected to the plugin system. This is only of limited benefit to a reverse engineer who wants to use both, visual inspection and code/plugins, especially when facing large designs. With the new event system the GUI can react to changes in the data model, as well as associated data, regardless of the source, and display the new information accordingly.
Submodule Partitioning System. While textual output is a primary method of reporting analysis results, HAL allowed a reverse engineer to apply a color to a net or gate in order to create visual groups. However, these colors were isolated GUI features and could not be created or accessed in plugins or from python code.
We replaced the isolated coloring system with a new system that allows the user to group identified gates or nets into logical modules, e.g., an identified transceiver circuit. These so called submodules are persistent and can be created and accessed via code and interaction with the GUI. To preserve the visual separation, each submodule is rendered in a separate color to allow for intuitive visually distinction. Since a gate or net can belong to multiple submodules, colored squares on each gate represent all submodules a gate is a member of. In addition the hierarchy of submodules can be changed to determine which submodule's color has higher priority during rendering. We plan to also include collapsing of submodules into single abstract elements in the future.
Python Integration. The powerful plugin system of HAL enables implementation of arbitrary algorithms in C++. However, a reverse engineer often has to manually explore the adjacent area of identified modules. Going through the rewrite/compile/debug cycle of C++ development can make this process tedious and slow. Therefore, the original version of HAL featured a python console, i.e., the user could instead of executing a plugin switch to a python console. The intuitive and, more importantly, interpreted nature of python allows for dynamic execution of HAL's features, circumventing the necessity to develop plugins for every small step.
The downside was a clear separation of HAL runs between python console usage, GUI invocation, and plugin calls. We have integrated the python console into the GUI together with a python code editor, allowing for interactive development and execution of scripts (cf. Figure 4, right side) without the necessity to restart HAL for each use-case. To further improve interoperability, the python context is connected to the underlying graph model of the netlist and all operations are directly executed on the data model. This allows for interleaved usage of C++ plugins, python scripts, and user actions in the GUI.

Educational & Research Material: Hardware Reverse Engineering Projects
With our enhancements to HAL, we were able to accompany our lectures with practical reverse engineering. At the same time, we created a platform which enables reproducible evaluation of human processes by means of several reverse engineering projects. The purpose of these projects is twofold. On the one hand the projects were developed to accomplish educational goals teaching students how to practically apply various reverse engineering techniques. On the other hand they had to be implemented as research materials to analyze human behavior in realistic hardware reverse engineering scenarios. To investigate if the twofold purpose of the projects was successful, a pilot course has been conducted [28]. The results of the educational evaluation showed the successful implementation of the projects as educational and research materials. Nevertheless the analysis revealed indications for revising aspects of the single projects. In the following the revised projects which will be used for our upcoming larger studies are presented.

Project 1 -Introduction to
HAL. An introductory project helps students to understand the various features and the functionality of HAL. It imparts knowledge about different gate types and their functionalities, and how HAL can be incorporated for the reverse engineering process. Dedicated HAL features like automated generation Boolean functions -or rather Binary Decision Diagrams (BDDs) -to analyze the logic of gates are introduced. Moreover applying different algorithms from graph theory, e.g., identifying strongly connected components, are included in project 1, which will support students in solving hardware reverse engineering tasks. For that matter a simple cipher has been designed consisting of a substitution-permutation-network.

Project 2 -FSM Reverse Engineering.
In this task an FSM circuit has to be identified in the sea of gates and the transition logic has to be reversed. The challenge in this task is to find the FSM in the netlist with several thousand gates. The task makes heavy use of novel features of HAL, like the submodules to group parts in the netlist that belong together, as described in Section 4.3. The detection of the FSM has become more realistic by finding false positive results when it comes to identifying possible FSM circuits. The challenge for the students is to analyze the candidates in more detail. Algorithms from graph theory, supporting the search, can directly be executed via HAL plugins. Once the component has been identified the Boolean functions of the state transition logic has to be explored in order to retrieve the states and transitions. Live exploration can be done easily with the integrated Python interface to control HAL (cf. Section 4.3).
What psychological insights will we gain?
• First reference point of how long students need to solve the FSM reversing task in total. • Analysis and description of students' mistakes during the FSM reversing task. • Analysis of how much extra time effort students spent on solving mistakes.

Project 3 -Obfuscated FSM.
In Section 2.2 several obfuscation techniques to protect the control path -and thus the FSM -have been described. Therefore, Project 3 incorporates reversing of an FSM based obfuscation technique named HARPOON [5], as shown in Figure 6. HARPOON inserts a second FSM (red) to protect the original FSM (blue). The inserted FSM part has to be traversed in a certain way, using a specific input sequence, called the enabling key. Every other input sequence than the enabling key will not lead to the original FSM thus rendering the hardware design unusable for unauthorized parties.  Figure 6: Obfuscated FSM using HARPOON [5] In this project the students need to apply the methods they learned in Project 2 in order to detect the FSM components in a gate-level netlist. In addition, the challenge is to circumvent the implemented obfuscation HARPOON [5]. In summary, two different attacks have to be performed: First, the states and transitions of the FSM have to be reversed to disclose the enabling key. Second, students will learn how to reverse the obfuscation and manipulate gate-level netlists to patch the initial state of the FSM [11].
What psychological insights will we gain?
• Analysis of how much time students required to solve the second FSM reversing task in total. • Comparison with reference point of total required time in Project 2. • Analysis if students have solved Project 3 more efficiently (e.g., shorter solution time, less mistakes, recovering from mistakes faster).

Project 4 -AES.
Project 4 combines dynamic and static netlist reverse engineering by asking the students to extract a fixed key from an AES design. Therefore the following steps have to be performed: First the AES S-Box has to be identified in the netlist and manipulated using HAL in order to weaken the AES (static part). Secondly, the now manipulated netlist can be used for dynamic analysis. Due to purposeful manipulation of the netlist, simulations of the design allow for extraction of the fixed key [25].
What psychological insights will we gain?
• Analysis and description of used hardware reverse engineering strategies in the most complex task of the study. • Analysis and comparison of total solution time, made mistakes and time for solving mistakes. • Descriptive analysis of how students recognized patterns and structures in finding the relevant candidates for extracting the AES key.

CONCLUSION
In this paper we highlighted serious security threats evolving from attacks based on hardware reverse engineering and identified the need for novel obfuscation methods impeding such attacks. Based on the inseparable connection of technical aspects and human factors for conducting hardware reverse engineering we derived the necessity for obfuscation methods considering both technical aspects and the behavior of human analysts. To adequately capture human reverse engineering strategies we significantly enhanced our hardware reverse engineering tool HAL. In the next step of our interdisciplinary research, we focused on the exploration of human factors in hardware reverse engineering by analyzing their learning processes and identifying common mistakes in four realistic scenarios. Therefore, we methodically include both longitudinal and cross-sectional analyses to understand and characterize hardware reverse engineering processes of individuals at different levels of expertise. Future studies will generate deeper insights about underlying human processes, which may allow for deduction of quantification metrics, eventually enabling better obfuscation methods.