P2FAAS: Toward Privacy-Preserving Fuzzing as a Service

Global corporations (e.g., Google and Microsoft) have recently introduced a new model of cloud services, fuzzing-as-a-service (FaaS). Despite effectively alleviating the cost of fuzzing, the model comes with privacy concerns. For example, the end user has to trust both cloud and service providers who have access to the application to be fuzzed. Such concerns are due to the platform is under the control of its provider and the application and the fuzzer are highly coupled. In this paper, we propose P2FaaS, a new ecosystem that preserves end user's privacy while providing FaaS in the cloud. The key idea of P2FaaS is to utilize Intel SGX for preventing cloud and service providers from learning information about the application. Our preliminary evaluation shows that P2FaaS imposes 45% runtime overhead to the fuzzing compared to the baseline. In addition, P2FaaS demonstrates that, with recently introduced hardware, Intel SGX Card, the fuzzing service can be scaled up to multiple servers without native SGX support.

Introduction
Fuzzing-as-a-service (FaaS) is an emerging paradigm to accelerate the adoption of a popular bug finding technique, fuzzing. FaaS services, such as Google OSS-Fuzz [8], Microsoft Security Risk Detection [21], and Fuzzbuzz [5], have been designed to alleviate technical and operational burdens of integrating fuzzing as part of software development: e.g., incorporating a well-known fuzzing driver, such as AFL [18], that provides various mutation strategies for the software under testing, and scaling it to a large number of servers on demand.
However, such benefits come with serious privacy concerns of the software, hindering the wider adoption of the fuzzing technique. First of all, in the current FaaS model, developers should trust the cloud provider, its underlying infrastructure, as well as operators; it means that pre-released software under active development has to leave out of the developer's complete control. Second, since the fuzzing techniques are continuously discovering 0day vulnerabilities, malicious (i.e., compromised or curious [4]) cloud providers would face some incentives to monetize the security bugs without disclosing them to the software developers. Third, such limitations are the concerns of not just the users (i.e., software developers) but cloud providers as well: the cloud providers should spend more operating costs to manage the security of the infrastruc-ture and the users' software, as well as to avoid the damage of the users' reputation when leaked.
We propose a new ecosystem that preserves users' privacy while servicing fuzzing on the cloud, shortly P 2 FAAS. The key idea is to utilize a modern, commodity trusted execution environment (TEE), called Intel SGX, as the root of trust. It not only helps FaaS' users eliminate the cloud provider from the trust domain, but also helps the FaaS providers focus on taking advantage of the economy of scaling without worrying about users' privacy. We believe fuzzing can be a killer application of SGX for two reasons: 1) fuzzing is a CPU intensive task avoiding the current memory limitation of SGX, and 2) when the active working set is small, SGX provides near-native performance so that developers do not have to trade performance-fuzzer's great strength-off for privacy.
This paper attempts to draw a picture of an end-to-end ecosystem of the SGX-enabled FaaS that addresses the privacy concerns of end-users as well as the scalability and early adoption of the fuzzing service. P 2 FAAS provides a toolchain that assists developers to package their software for testing in a privacy-preserving manner ( §4.2) securely scaling to multiple servers as necessary ( §4.3), and helps the cloud operators to adopt it even on a legacy server with an extension card, Intel SGX Card ( §4.4). We make P 2 FAAS as an open-source project to inspire and direct current FaaS providers to foster the fuzzing techniques for broader audience in a convenient and privacy-aware manner. Summary. This paper makes the following contributions: • We propose an end-to-end ecosystem, P 2

Background and Related Work
Fuzzing. The idea of fuzzing is providing randomly mutated inputs to a program that aims for triggering abnormal behaviors such as crashes representing potential bugs. This process allows fuzzing to execute automatically and detect bugs with high accuracy. To improve the effectiveness of fuzzing, one direction is to increase the throughput of fuzzing. Approaches for this direction include scaling up the fuzzing with distributed machines [7] and designing specialized system support [30]. Another direction is to improve the strategy of the seed selection; that is, inputs derived from a seed that trigger more execution paths of the program usually lead to more bug findings.
Fuzzing as a service. Instead of physically owning and maintaining multiple machines for fuzzing, end-users now have an option to fuzz their applications on the cloud via "fuzzing as a service" (FaaS). The idea of FaaS has service providers to set up the fuzzing infrastructure, typically involves numbers of machines, on the cloud such that end users can fuzz their applications with a pay-as-you-go model. The existing service providers include cloud providers themselves (e.g., Microsoft and Google) and third parties (e.g., Fuzzbuzz and Fuzzit). Although FaaS makes fuzzing more accessible and successfully finds thousands of bugs in real-world applications, privacy concerns arise.
Intel SGX Applications.
Communities have proposed several SGX-based solutions for preserving privacy in existing applications. The examples include network functions [26,24], anonymity network [14,13], and machine learning [22,11]. To the best of our knowledge, P 2 FAAS is the first work that uses SGX to address the privacy problem with FaaS. Intel SGX Card. Although quickly becoming a general security feature in recent desktop-focused CPUs, SGX receives relatively small support in server-class CPUs that cloud platforms typically adopt. To fill this gap, Intel has recently introduced new hardware, the Intel SGX Card. Intel SGX Card is a re-configured graphic card that consists of three independent, SGX-capable CPUs. Similar to a graphic card, the Intel SGX Card is pluggable to multi-socket server CPUs (connected via PCIe interfaces), and each CPU can have up to four such cards. As a result, the cards allow both an SGX-and a none-SGX-capable CPU to have additional SGX support with a high resource density. Moreover, opposed to buying new SGX-capable machines, adding the cards to existing ones is more cost-and space-efficient. These advantages make the Intel SGX Card an optimal, practical option in response to the growing demand of SGX support in the cloud [6,19].

Threat Model
The setting of P 2 FAAS involves three parties, including a cloud provider who offers SGX-capable platforms, a service provider who sets up fuzzers with enclaves in the cloud platforms and offers fuzzing as a service, and an end-user who uses the service to fuzz her application. Among these parties, P 2 FAAS assumes the cloud provider is the only untrusted one-either being compromised or simply because of the other parties do not fully trust-who aims for obtaining information about the fuzzer and the application which both are proprietary. Having full control over the cloud platforms, the cloud provider can intercept all the incoming and outgoing network traffic. Further, the cloud provider can freely analyze an initial program binary to be running inside an enclave. However, our model assumes the program binary does not contain memory corruption vulnerabilities that allow for control-flow hijacking and memory leaks. Although SGX ensures the confidentiality of the enclave during its runtime, the cloud provider may still learn information about the enclave based on observable behaviors such as crashes. Our model considers side-channel attacks [9,31,16] against SGX as out of scope. However, existing side-channel mitigations [27,10,23] are applicable to P 2 FAAS.

Goals
1 Privacy-preserving. P 2 FAAS considers two types of privacy: a fuzzing instance and its runtime behavior. The fuzzing instance includes a target program and a fuzzer that provides mutated inputs. In addition to prevent the cloud provider from accessing the program and the fuzzer, P 2 FAAS also aims for protecting the inputs, especially for ones that trigger crashes. Leakage of such inputs indirectly discloses the vulnerabilities of the program. Moreover, P 2 FAAS aims to prevent the cloud provider from directly inferring the observable runtime behavior, more specifically, crashes. The occurrence of crashes directly affects the reputation of end users (i.e., indicating the number of bugs in the program). 2 Performance. Runtime performance directly contributes to the quality of fuzzing services (i.e., number of bugs found). Poor performance reduces the quality of service and weakens the motivation of using the service. Therefore, naïvely trading performance for privacy in the case of fuzzing is not acceptable. P 2 FAAS aims to maintain the fuzzing performance while achieving privacy preservation ( 1 ). 3 Deployability. P 2 FAAS aims for being deployable to existing cloud platforms. One aspect for this requirement is that platforms should provide SGX support that P 2 FAAS depends on. For platforms that already provide native SGX support, such as Microsoft Azure [20] and IBM Cloud [12], P 2 FAAS is directly deployable. For platforms provide SGX support via additional hardware (e.g, Intel SGX Card), P 2 FAAS should also accommodate to such environment. The other aspect is that P 2 FAAS should allow for easily adoption; that is, the service provider requires minimal effort to scale up the fuzzing service across multiple machines regardless of the type of SGX support (native or non-native).

Workflow
We provide a detailed walk-through of steps to achieve a complete service cycle with P 2 FAAS, as shown in Figure 1. 1 Fuzzing toolchain. The service provider develops a complete fuzzing tool-chain in order to provide fuzzing service. Those tools include, but not limited to, a fuzzer and corresponding public APIs, a compiler that generates binaries conform to SGX programming practices, and a loader that loads both the fuzzer and the target program into SGX enclave memory. Although parts of the fuzzing toolchain, the compiler and loader are not privacy sensitive but publicly available and verifiable, serving as trusted anchors to assist further provisioning of the fuzzing instance into trusted SGX enclave memory.
2 Service tool provisioning. The toolchain loader developed by the service provider will be provisioned to SGX enclaves by cloud providers before fuzzer APIs are released to the public. Once provisioned, the cloud provider will be ready to accept fuzzing instance submissions and serve endusers with fuzzing service.
3 Applying fuzzer API. The prospective end-user integrates fuzzer APIs released by the service provider into the target program during development, and compiles the program using the toolchain compiler. Usages of fuzzer APIs vary across different fuzzers and is up to the service provider to specify. 4 Fuzzing instance provisioning. The target program binary will be submitted to the cloud provider after being compiled by the end-user. Meanwhile, the service provider will be notified and the corresponding fuzzer binary matching the fuzzer API usages of the target program will also be submitted to the cloud provider. The toolchain loader will then load the fuzzer binary and the target program binary into individual SGX local enclaves when both are present. 5 Fuzzing instance execution. After both the fuzzer and target program binaries are loaded into SGX enclaves, execution of the fuzzing instance starts. Necessary service information is transmitted constantly via the trusted communication channel between local enclaves. Specifically, the fuzzer generates and passes numerous inputs to the target program for execution, and the target program passes back essential feedback information to the fuzzer for consumption (e.g., coverage feedback, crashes). 6 Fuzzing results. Finally, the fuzzing service periodically sends fuzzing results (e.g., crashes triggered) back to the end-user via a trusted communication channel between the cloud provider and the end-user. The trusted communication channel can either be established as TLS using SGX Remote Attestation, as suggested by Intel [15], or other methods specified by the providers.

Oblivious Crash
The fuzzing instance is naturally protected since it entirely resides in SGX enclave memory. However, program behaviors especially crashes, require additional protection mechanisms as the cloud provider is directly aware of such events. P 2 FAAS resolves this challenge by introducing a technique called oblivious crash. Whose name adopted from oblivious memory, oblivious crash hides the actual crash behavior from observers. That is, by randomly generating fake crashes from the target program after fuzzer instrumentation, the crashing behavior is normalized and external observers cannot distinguish real crashes from fake ones.
Not only that, P 2 FAAS also utilizes fakes crashes as heartbeat messages to attest availability and legitimacy of cloud provider. On one hand, fake crash results with extended time interval indicates reduced network condition, and failing to deliver fake crashes periodically indicates lost of service availability. On the other hand, degraded fake crash heartbeat quality might also indicate potentially malicious operations are performed, e.g. the cloud provider pauses the execution and attempts to distinguish fake crashes. P 2 FAAS provides the end user with a handful way to sense such situations upon experiencing different heartbeat behaviors, and react accordingly.

In-Memory Corpus Management
One of the significant performance bottlenecks of SGX application is OCalls. To minimize OCalls while fuzzing, P 2 FAAS keeps the corpus and coverage map inside the enclave as long as the memory can tolerate. From the initial fuzzer setup, P 2 FAAS reads in every content of the shared corpus directory. Afterwards, corpus related operations such as mutating and pruning are done inside the memory. If the memory is overwhelmed, or the corpus status is significantly updated, the in-memory corpus is synced with the shared corpus directory. Ultimately, the shared corpus directory selectively holds the corpus that maximizes the code coverage.

Platform-independent Architecture
Similar to traditional fuzzing in parallel [30], P 2 FAAS fuzzing instances can easily run on multiple SGX-capable nodes in parallel to scale up execution speed. However, as shown in Figure 1, there are two types of platforms within cloud providers: SGX-capable platforms and SGX-incapable ones. Likely, instances reside in different cloud providers consist of a mixture of those two types of platforms.
In order to deploy P 2 FAAS fuzzing instances on multiple cloud nodes in parallel and achieve the performance scalability without platform restriction, P 2 FAAS considers Intel SGX Card as the most economical and pluggable SGX enabler for SGX-incapable platforms. Mutated inputs are shared via the network across all P 2 FAAS fuzzing instances (shown by 5 in Figure 1), either hosted on natively SGX-capable platforms, or hosted on SGX nodes inside Intel SGX Cards. Therefore, P 2 FAAS achieves platform-independent deployability and scalability by treating each SGX-incapable platforms as several (3-12) SGX-capable nodes enabled by Intel SGX Cards. We further show that distributing P 2 FAAS to SGX nodes enabled by Intel SGX Card achieves the same performance scalability as by multiple natively SGX-capable instances in section §6, proving that Intel SGX Card is the key to maximize P 2 FAAS deployability and scalability.

Implementation
We implemented the prototype of P 2 FAAS with roughly 1,000 lines of C code on top of existing open source projects (SGX-Shield [25] and libfuzzer [17]). The prototype includes a fuzzer, a customized toolchain, and an in-enclave loader.
Fuzzer. For simplicity, our fuzzer adopts the model of libfuzzer, which is a part of the target program, allowing the program to fuzz itself. We leave the fuzzer decoupling from the program as future work. Our fuzzer is tailored to dealing with the limitations of SGX (e.g., limited memory and no syscall capabilities). For example, to implement the mutation engine without OCalls, we use sgx-rand for randomness entropy. Ultimately, the fuzzer exports a call-back API similar to the fuzz-driver in libfuzzer.
Toolchain. P 2 FAAS service user should port their code with the provided call-back API; that is, compiling the program with our toolchain (Clang/LLVM 4.0) with a new sanitizer option (-fsanitize=sgxfuzz) for the LLVM pass. The compiled program automatically generates both coverage information and crash reports during the fuzzing. Our implementation uses 4KB edge coverage map similar to AFL. To optimize the runtime performance (i.e., minimizing the number of ECalls and OCalls), we maintain the seed corpus inside the enclave and only synchronize it with the file system when needed.
Oblivious crash. To implement oblivious crash, we added a fake-crash generator as a part of the LLVM pass. The Fake-crash generator randomly chooses a time between the pre-configured interval. If a real crash does not occur until the configured time period, the instrumented program triggers crash. The crash is caused by invalid memory access to random address (using the sgx-rand API). Upon crashes (real or fake ones), the instrumented program sends out the crash report. To further hide the crash information, we can encrypt the report.
Cloud provider. For P 2 FAAS cloud provider, we setup an SGX host (Intel i7-6700K) and a No-SGX host (Intel Xeon E5-2620) with two Intel SGX Cards. We pin the fuzzer to each of cores for distributed fuzzing. For corpus and coverage synchronization, we use network file system (NFS) interface.

Evaluation
In this section, we evaluate the achievement of privacy preservation, the usability and performance characteristics of P 2 FAAS. Experiment setup. We ported three fuzz-drivers: (i) typical toy example for libfuzzer/AFL (byte-to-byte string match), (ii) arithmetic expression evaluator (30 LoC), and (iii) C-ARES DNS API from Google fuzzer-test-suite (1,225 LoC). Our evaluation numbers are based on an average of ten iterations.

Privacy Preservation
While the fuzzing instance is protected by relying on SGX security guarantees, P 2 FAAS target program raises random fake crashes based on a configurable normalized crash frequency to obfuscate its runtime behavior (oblivious crash). Figure 2 shows the outcome of a normalized 3 crashes per minute. Specifically, the frequency of real crashes starts with a spike and decreases as time progresses. Such runtime behavior is normalized by injecting fake crashes with an inverted frequency distribution. The resulting observable crash behavior is represented by Real+Fake in Figure 2, where the crash frequency appears stable over time. By applying such techniques to normalize crash frequencies, real crashes are indistinguishable from fake ones, thus protecting the runtime behavior of the target program.

Usability
One of the major challenges affecting P 2 FAAS usability is porting the target program into SGX, as SGX does not natively support system calls. To evaluate P 2 FAAS with our libfuzzer-based implementation, 15 OCalls are added in the fuzzer to accommodate SGX environment. For end users, depending on the target program and fuzzer interface, number of required OCalls can significantly differ. In our evaluation, we selected target programs that do not invoke system calls to avoid adding OCalls.
For the toy and arithmetic parser examples, the entire programs operate based on memory and arithmetic operations. In the case of fuzzer-test-suite example, the parser with vulnerability (CVE-2016-5180 [1]) does not require to invoke any system call. However, depending on the fuzzer interface and program size, non-trivial porting effort could be involved. We note that these porting efforts can be relaxed with existing techniques [29]. 0k   200k   400k   600k   800k   1000k   0  5  10  15  20  25  30  35  40  45  50 Exec/sec # core (w/ Intel SGX Card) Figure 3: P 2 FAAS performance scaling with Intel SGX Card.

Performance Characteristics
Scaling with Intel SGX Card. To demonstrate the scalability of P 2 FAAS, we used two Intel SGX Cards to setup a 48-core distributed P 2 FAAS environment. Fuzzers running on each core share the corpus and coverage information via network file system (NFS). Using toy fuzz-driver, we were able to achieve near one million executions per second as shown in Figure 3. CPU overhead. To measure the CPU overhead of P 2 FAAS, we show a comparison result with three configurations: (i) running the fuzzer without actually using the SGX feature at all (no-sgx), (ii) running the fuzzer inside enclave with SGX simulation mode (sgx-sim), and (iii) running the fuzzer inside enclave with SGX hardware mode (sgx-hw). In particular, we measured the execution time of one million executions. Numbers in Table 1 are the average of ten iterations.
The evaluation results show that SGX environment imposes around 45% performance overhead. We suspect this is due to the less optimized cache/TLB implementation. Memory overhead. Memory usage is another potential cause of performance degradation of P 2 FAAS under SGX environment. We measure the resident set size (RSS) with and without P 2 FAAS using pmap to analyze the memory overhead.
Regardless of the target program, the evaluation results give a memory overhead of around 2MB. The memory overhead is mainly caused by the in-memory corpus data structure, coverage bitmap, added fuzzer runtime code, and so forth. Such a small memory overhead is ideal for SGX environment as the runtime memory consumption of the fuzzing instance is more likely not to exceed SGX EPC size limitation, thus avoiding the long-worried performance degradation caused by SGX demand paging mechanism.

Discussion and Future Work
Inferring information from fuzzing inputs. Using SGX allows P 2 FAAS to protect the target program and its fuzzing inputs against the cloud provider. To further prevent the service provider from accessing the program, P 2 FAAS puts the program and the fuzzer in separate enclaves. However, because the program still relies on the fuzzer to provide inputs, the service provider may be able to infer the information about the program from the inputs. At this point, P 2 FAAS do not protect programs from such threats. We leave it to future works. Programs with large working set. Our evaluation test cases used a reasonably small amount of memory, thus fitting an enclave. Currently, size of SGX Enclave Page Cache (EPC) is limited to 128MB. Instances with larger working set that exceeds the size limitation will incur the SGX demand paging mechanism, which is extremely expensive, slowing down the system on average by 5 times [28]. This is harmful to the service quality of P 2 FAAS. We leave the solution to foreseeable future SGX support for larger EPC size.

Conclusion
Motivated from the rapidly emerging FaaS services, we propose P 2 FAAS to provide privacy on top. Our design demonstrates that SGX is a promising underlying technology for P 2 FAAS without harming usability and performance. As a prototype, we implemented a P 2 FAAS infrastructure based on SGX-enabled host and none-SGX host with SGX Card extension. Our implementation and evaluation demonstrates that P 2 FAAS is scalable and deployable in practice.


extfieldsofstudy:
Medicine
Chemistry
Materials Science
Physics

text:
COMBIgor: data analysis package for combinatorial materials science

Combinatorial experiments involve synthesis of sample libraries with lateral composition gradients requiring spatially-resolved characterization of structure and properties. Due to maturation of combinatorial methods and their successful application in many fields, the modern combinatorial laboratory produces diverse and complex data sets requiring advanced analysis and visualization techniques. In order to utilize these large data sets to uncover new knowledge, the combinatorial scientist must engage in data science. For data science tasks, most laboratories adopt common-purpose data management and visualization software. However, processing and cross-correlating data from various measurement tools is no small task for such generic programs. Here we describe COMBIgor, a purpose-built open-source software package written in the commercial Igor Pro environment, designed to offer a systematic approach to loading, storing, processing, and visualizing combinatorial data sets. It includes (1) methods for loading and storing data sets from combinatorial libraries, (2) routines for streamlined data processing, and (3) data analysis and visualization features to construct figures. Most importantly, COMBIgor is designed to be easily customized by a laboratory, group, or individual in order to integrate additional instruments and data-processing algorithms. Utilizing the capabilities of COMBIgor can significantly reduce the burden of data management on the combinatorial scientist.

Introduction
High-throughput experimental (HTE) research, which consists of synthesis and characterization of specimens with spatially varied processing parameters, has been established as an effective strategy for exploring and optimizing materials. These methods, also known as combinatorial science, have proven useful for discovery and optimization of materials for diverse applications, including catalytic materials, 1 electronic and functional ionic materials, 2 polymer coating materials, 3 sensing materials, 4 and biomaterials. 5 With the success of combinatorial science, a diverse set of high-throughput synthesis and characterization in-struments has been developed and demonstrated, 6 facilitating rapid generation and analysis of HTE materials data.
The ability to input and analyze data is typically the rate-limiting process in HTE methods. 7 The HTE researcher must process, analyze, and correlate a large amount of data in order to understand the variations within each combinatorial library, then understand how trends correlate between libraries. This process is inefficient, as the experiments are put on hold so that scientists can wade through the data before continuing. Programmatic task handling ensures that effort is not unnecessarily wasted on routine tasks, such as loading, sorting, labeling, and tracking data. Accessible data and flexible analysis algorithms are needed for processing and analyzing various types of data and effectively interpreting results.
There have been several combinatorial software packages presented in literature. One example is CombiVIEW, 8 a MatLab package for the analysis and visualization of x-ray diffraction patterns generated from combinatorial approaches. Another package was written to improve HTE data organization and sharing for sensor materials. 9 Developments like these have significant depth. They handle one type of data or perform one specific task exceptionally well. The lacking element is breadth, the ability to handle many different types of HTE data in many different ways, so that the data can be more efficiently stored, compared, shared, and presented. Thus, a package is needed that has sufficient breadth, while being flexible enough to enable the user to develop depth for their own processes and data types.
Here we present COMBIgor -a data analysis package for Igor Pro intended to aid in the management, processing, and cross-correlation of multi-dimensional data sets collected by mapping combinatorial libraries on spatially-resolved measurement instruments. The core functionality was intentionally designed to enable flexibility in data types and experimental setup, while enabling the user to expand usability by adding new measurement instruments and new data analysis algorithms. Below, we present an overview of the functionality and architecture of the open-source COMBIgor package. This package can reduce the burden of data management of combinatorial experiments, thus encouraging its adoption by combinatorial researchers, groups, and institutions.

COMBIgor History
Development of this software package started in the early 2000s by materials scientists at the National Renewable Energy Laboratory (NREL) in Golden, Colorado alongside their emerging thin-film HTE methodology. From 2000 to 2010, several software add-ons (referred to as "procedures" in Igor Pro) were developed in order to load, store, and display combinatorial data from individual instruments, such as a Bruker D8 Discover x-ray diffractometer (XRD) and MaXXi 5-pin x-ray fluorescence (XRF) spectrometer. In 2010, this collection of procedures was extended to include custom optical and electronic property mapping instruments. 10 Most importantly, these different procedures were integrated into a package allowing processing and cross-correlation of different types of data, as cited in two published manuscripts. 11,12 Since that time, the package has continued to grow as new and continuing users have added new instruments and new functionality, and has since been used to process data and generate figures for a large number of publications. Here, we intentionally cite the publications in which this package was mentioned, but not cited, so that future readers of these publications can find the present manuscript and better understand the software package used. 10, Around 2016, the authors started considering improved methods of data management, usability, and visualization, as well as the potential for the package to be extended by outside users and developers. In the two years that followed, the collection of Igor procedures was restructured and optimized into an interactive and flexible Igor package for loading, storing, and processing combinatorial data, and was given the name COMBIgor (COM-bee-gore).
This newer version has already been used for data analysis and preparation of figures in several recent manuscripts. [49][50][51][52][53][54] Based on this history, COMBIgor brings the experience of almost 20 years of software development at NREL for handling and processing combinatorial data. Additionally, it has been integrated with public data repositories to enable cutting-edge data transparency and openness. 55

COMBIgor Terminology
The terminology utilized within the COMBIgor package is meant to reflect the common descriptors used in HTE literature. A list of the most important terms is shown in Table 1.
For example, it is important to understand the concept of a library in the COMBIgor package. A library is a specimen with spatially-varying processing conditions, such as substrate temperature or plasma flux. A collection of libraries constitutes a project. Libraries are measured on a mapping grid -a discretized, spatial arrangement of points from which data are collected on a set of instruments. A single point of the mapping grid, which corresponds to a specific location on a library, is called a sample. This language is used extensively in COMBIgor for referencing HTE specimens and their corresponding data.
One of the most important conceptual advancements of COMBIgor is that it categorizes the data based on dimension, scope, and type, regardless of its source, allowing for a greater level of abstraction and code reuse. Data scope can include a single sample or all samples on a library; data can be numeric or textual; and the numeric data dimension can be scalar, vector, or matrix. For example, x-ray diffraction patterns and optical absorption spectra are both numeric vector data types comprising one-dimensional data collected for each sample on the library. Other important features within COMBIgor are add-ons, which are categorized as either instruments or plugins. An instrument is a source of synthesis or characterization data from combinatorial experiments, such as XRD or XRF. A plugin is a specific set of operations that process data within COMBIgor. Examples include background removal or applying mathematical operators.

Distribution and Use
At time of publication, the COMBIgor package is housed and maintained through a pub- COMBIgor expands after publication of this manuscript.
After installation (described in section 3.1), the COMBIgor package exists within the Igor Pro environment and can be activated at any time through the native Igor Pro "Data" menu.
Once activated by the user, the COMBIgor procedure files are mounted into the experiment, the user menu is created, and the add-ons selector is displayed. After the desired add-ons are selected and the project (with its mapping grid) is defined, the consequent workflow varies depending on the needs of the user. Generally, it involves repeatedly loading, processing, plotting, and exporting various types of HTE data. A snapshot of COMBIgor package elements within the Igor Pro environment can be seen in Figure 1. More details on the COMBIgor workflow are provided below.

Workflow
The generalized workflow for using COMBIgor is presented in Figure 2. After COMBIgor package activation, the first six steps ("Setting Up COMBIgor" in Figure 2) define the project space, activate the desired add-ons, and initialize the parameters necessary to use them.
A project is defined by a common combinatorial mapping grid for all libraries contained within that project. The user must then define loading parameters related to the individual instruments, such as data type names, location of data in the files, and processing-related variables. Unlike the mapping grid, these can be redefined during the course of the project lifetime, although these steps are only required when starting a new project. Subsequently, data are managed through a project, library, and data type hierarchy. The last five steps ("Analysis of combinatorial data", Figure 2) are executed repeatedly as the scientist loads and analyzes experimental data. At this stage, the user loads combinatorial data collected from instruments using the standard mapping grid into the user-defined library folders. Once data are loaded into the COMBIgor project, they can be analyzed and visualized using instrument-specific routines, plugins, or native Igor Pro routines. The num-ber of libraries loaded, instruments utilized, and processing steps will vary depending on the project and the user, as reflected in the loops of Figure 2. During this process, COMBIgor data loading and processing events are logged into a searchable log book. At any point, the raw data can be exported for distribution into tab-delimited text files for loading into additional COMBIgor experiments or other programs, as needed. This exporting capability is particularly important for multi-institutional projects, such as the High-Throughput Experimental Materials Collaboratory. 57 All figures generated in COMBIgor can be further modified by the user and exported for use in presentations and manuscripts.

Functionality
To demonstrate the functionality of COMBIgor, data from combinatorial Al 1−x Cr x N thin film sample libraries are used in the following sections. This material system has been recently investigated for structural evolution, electrical, and optical properties. 49 The Al 1−x Cr x N heterostructural alloys were deposited from elemental Al and Cr targets in the presence of nitrogen via radio frequency co-sputtering. The compositions of these combinatorial libraries were determined by x-ray fluorescence (XRF), structures by x-ray diffraction (XRD), thicknesses by step-edge profilometry, optical properties by optical transmission/reflection spectroscopy, and electrical conductivity by 4-point probe sheet resistance measurements. More details regarding the deposition or characterization instruments can be found elsewhere. 50

Installation and Preferences
The installation process for the COMBIgor package is designed to be as simple as possible.
An installer experiment file (COMBIgor_installer.pxp) is included in the package folder.
When opened, this file will install the package contents and any necessary Igor Pro extensions.
The user can install the package with a single click from the COMBIgor menu. The installer will preserve a copy of any COMBIgor package files that predate the current installation to protect any modifications a user has made to a previous package version. After installation, the user is directed to COMBIgor activation in the data menu, as well as the COMBIgor help menu and the included tutorial. In addition, we recommend that before using COMBIgor, the users familiarize themselves with the built-in Igor Pro functionality by taking the guided tour from the Igor Pro help menu.
The COMBIgor package has a built-in user preference system. The user can select a set of COMBIgor preferences, including setting a default import or export folder location, controlling the "kill" behavior of windows generated by Igor (e.g. figures, tables), and generating example programmatic call lines for operations performed by COMBIgor in the command history. The package can also generate standard plots upon loading data from instruments and automatically adjust font styling of control panels and plotted figures.

Instruments and plugins
The needs of HTE vary greatly between projects, users, and laboratories. If a single package contained all possible data loading, processing, and visualization functionality applicable to HTE, the package would be too complex to be efficiently navigated. For this reason, the COMBIgor package allows each user to select only the optional capabilities (add-ons) that they plan to use. In addition to keeping the package streamlined, instrument and plugin add-ons provide a mechanism for individual user expansion and laboratory-wide instrument collections.
The lists of the base instruments and plugins included at the time of publication of this manuscript are described in Table 2 and

Data Loading and Processing
Experimental data comes in many different file formats: some are stored as a simple text file, some are contained in proprietary file formats, and most are something in between. Taking advantage of the data-handling features of Igor Pro, COMBIgor is able to import a wide variety of data formats. Data loading involves programmatically opening native instrument file types, transferring the data for each sample into an Igor Pro wave that matches the Masks, fits, and subtracts backgrounds from vector data, such as x-ray diffraction. DiffractionRef Imports, stores, and plots powder diffraction references exported from common diffraction programs and databases. VectorPeakFit Executes constrained peak fits on a library of vector data and extracts fit information.

DataSifter Plugin
Applies a conditional test and, upon true, performs some resulting action across every sample on a library. MathPlugin Performs simple mathematical operations to data across every sample on a library. Piemaps Plots library maps on which each sample marker is a pie chart of multiple scalar data types. TernaryPlotter Generates ternary plots with marker color and size determined by scalar or vector data. SolarCell Extracts solar cell performance parameters from current vs. voltage data.
dimensionality of the incoming data, and storing those waves in the appropriate project and library folders. Loading steps include assigning identifying information, such as project, library, sample index, data type, and data dimension, to the incoming data; creating the data storage in the correct location within the COMBIgor experiment; and populating that storage with the appropriate data. The load event is documented in the project log book to retain a record of the original data source.
In COMBIgor, data are typically loaded through instrument add-ons ( Figure 3). Additionally, data can be imported from other COMBIgor files as COMBIgor text files. Finally, COMBIgor has the ability to fetch data directly from online databases through application programming interfaces (APIs) and store it locally for comparison to other data types. This feature is demonstrated by loading data from the High Throughput Experimental Materials  Data processing is integral to scientific experimentation. For the HTE researcher, the diversity of processing tasks combined with the magnitude of collected data can create a daunting workload. For this reason, COMBIgor data is extremely easy to batch-process, and can be processed in various modes. There are four ways in which COMBIgor data can be processed: • Upon loading by an individual instrument procedure This is the simplest method, and is utilized for standard measurement processing routines. Example: Feature height extraction from profilometry scans of thin film samples.

• By COMBIgor plugins
This is the most powerful method for HTE with high density mapping grids, since the same process can be systematically applied to every sample on a library. Example: Peak fitting to extract the FWHM of an x-ray diffraction peak. See Figure 3 for more detail. •

Data Visualization
Once data are loaded, the COMBIgor package can produce publication-quality graphics.
One of the strengths of COMBIgor is its ability to plot data types of different dimensionality on the same plot. There are four modes of graphics generation: • Upon instrument-specific loading If the user selects the option in package preferences, COMBIgor can automatically generate formatted plots as data are loaded through the instrument procedures. Example: Programmatic generation of a normalized diffraction intensity heat map comparing all samples upon loading X-ray diffraction data (see Figure 4, part A).

• Using the COMBIgor Display panel
The COMBIgor display panel is capable of producing two-dimensional figures with up to four data type-dependent axes: horizontal axis, vertical axis, marker color, and marker size. Any data dimensionality can be handled by each of these axes. Example: The Display panel can be used to generate a map of material resistivity for an entire library, on which the marker size for each sample is determined by film thickness (see Figure 5, part A).

• Utilizing native Igor Pro functionality
Data organized within a COMBIgor project can be utilized directly by Igor Pro graphics functionality to produce fully customized, publication-quality graphics. Example: Adding a visible spectrum color indicator to the axis of a COMBIgor generated 2D absorption coefficient plot greatly increases the visual appeal of this graphic (see Figure 5, part B.)

• Visualization plugins
Additional plugins extend COMBIgor functionality to specialized plot types. Plugins can be created and added to COMBIgor to generate any graphic type Igor Pro can produce. Example: Using the TernaryPlotter plugin to analyze the distribution of peak area as determined by the VectorPeakFitting plugin (see Figure 4, part B).
The magnitude of data generated by HTE can lead to comparison graphics that contain thousands of data points. Graphics produced by the display features of COMBIgor are connected to a display wave specifically created for this visualization and distinct from the original data waves. The source information of the wave data (e.g. library name, data type, data dimension, sample range, etc.) can be accessed and easily saved. However, this also means that the plot has to be remade when the data are updated or reloaded (i.e. the plot is not updated automatically). When plots are created, they can be saved to the export folder and recreation macros can be generated. Closing the plot will trigger deletion of the attached display wave, thus minimizing data duplicity.

Data Access and Porting
COMBIgor data processing and storage is designed to be transparent and easy to access so that the user understands the process and can modify the approach at will. To this end, all COMBIgor data are stored within the folders of the Igor Pro experiment, as shown in  When selected from the project menu, the data are presented in the Igor Pro data browser.
Combinatorial HTE is often collaborative. Results are published in peer-reviewed journals that encourage submission of raw data used in the publication, either as supporting information or in a separate data repository. Journals that publish mainly data, such as Data in Brief and Scientific Data, are also becoming more common. 63,64 For all of these reasons, data stored in COMBIgor must be accessible beyond Igor Pro in a simple file format.
COMBIgor can automatically save any locally-stored data onto the hard drive in a simple tab-delimited text file format with a folder structure that matches the project. As mentioned above, these exported data can also be imported into a different COMBIgor project, while maintaining the library names and sample numbers.

Documentation
A major strength of the COMBIgor package is the breadth and depth of documentation, which is designed to be easily accessed at all stages of use and development. The documentation is compiled using the Igor Pro help file type, as well as in-line procedure comments.
The documentation exists at seven levels: • A beginner's tutorial is included in the help menu and presented upon installation for new COMBIgor users to learn the basics of instruments, plugins, and display features.
The package includes example data for each of the instruments, which are used over the course of the tutorial. While example data are specific to NREL, this is intended to build a conceptual understanding of the work flow.
• General information about the mapping grid, data type dimensions, plugin operations, and instrument details are included in a general help file which gives useful details about COMBIgor and can be accessed through the COMBIgor menu. 2. The Instruments folder contains a single procedure for each instrument. Each of these procedures contains the unique functions needed to load and process data from that instrument.
3. The Plugins folder contains a single procedure file for each plugin, which builds its panel interface and performs the data manipulation processes. Tasks performed by the package elements can be categorized into three categories: core functionality, data handling, and add-ons. For each of these three categories, the package elements that perform or support the sub-tasks are illustrated in Figure 6 and described in more detail below.

Core functionality
COMBIgor core functionality includes installing, mounting, initializing, using, and navigating the package. Installation of the package involves placement of two aliases into the Igor Pro user folders and installation of built in XOPs. These steps can be done manually or by the COMBIgor_installer.pxp experiment. After installation, the user can activate COMBIgor from the data menu, enabling the remaining core functionality. These functions handle the storage and retrieval of global variables necessary for the package as well as for each individual project. Additional global variables can be added by future developers. The core functionality builds and enables the Projects menu featured in Figure 1, which enables the user to navigate the data stored in COMBIgor, create new projects, libraries, and data types, and import and export data from the project.

Data handling
Data handling tasks within COMBIgor are the most robust, and therefore complex, functions.
These functions create data storage waves, move data in and out of these waves, and transfer data in and out of the Igor Pro experiment. In addition to data management, these functions perform the vital roles of displaying and processing data. Because directly inspecting data in an array with more than two dimensions is difficult, data are typically parsed and stored in one-or two-dimensional waves. Each data type is a separate wave and is categorized by the project, library, and data dimensionality. The storage waves are of the Igor binary and Igor text file types, which are compressed and stored in an Igor packed experiment file. This results in an efficient use of memory when compared to the native instrument files provided as example data. The most essential data used for the example experiment occupy 80% less memory when stored in an Igor Pro packed experiment and 50% less memory when stored as COMBIgor text data files.

Add-on packages
Add-on package elements are the most diverse functions due to the wide variety of tasks they perform. By default, COMBIgor has the functions needed to initialize and run any add-on included in the Plugins and Instruments folders. Each plugin and instrument listed in Table 3 and Table 2 has documentation for users; however, programmer documentation does not exist for these add-ons, as they are unlikely to be modified by individual users (with the exception of the ExamplePlugin, ExampleVector, and ExampleScalar procedure files).

Object-oriented programming
COMBIgor was not written with an obvious object-oriented programming (OOP) architecture in mind. However, its hierarchical data structure can be explained in terms relevant to


extfieldsofstudy:
Computer Science
Medicine

text:
Multi-channel PINN: investigating scalable and transferable neural networks for drug discovery

Analysis of compound–protein interactions (CPIs) has become a crucial prerequisite for drug discovery and drug repositioning. In vitro experiments are commonly used in identifying CPIs, but it is not feasible to discover the molecular and proteomic space only through experimental approaches. Machine learning’s advances in predicting CPIs have made significant contributions to drug discovery. Deep neural networks (DNNs), which have recently been applied to predict CPIs, performed better than other shallow classifiers. However, such techniques commonly require a considerable volume of dense data for each training target. Although the number of publicly available CPI data has grown rapidly, public data is still sparse and has a large number of measurement errors. In this paper, we propose a novel method, Multi-channel PINN, to fully utilize sparse data in terms of representation learning. With representation learning, Multi-channel PINN can utilize three approaches of DNNs which are a classifier, a feature extractor, and an end-to-end learner. Multi-channel PINN can be fed with both low and high levels of representations and incorporates each of them by utilizing all approaches within a single model. To fully utilize sparse public data, we additionally explore the potential of transferring representations from training tasks to test tasks. As a proof of concept, Multi-channel PINN was evaluated on fifteen combinations of feature pairs to investigate how they affect the performance in terms of highest performance, initial performance, and convergence speed. The experimental results obtained indicate that the multi-channel models using protein features performed better than single-channel models or multi-channel models using compound features. Therefore, Multi-channel PINN can be advantageous when used with appropriate representations. Additionally, we pretrained models on a training task then finetuned them on a test task to figure out whether Multi-channel PINN can capture general representations for compounds and proteins. We found that there were significant differences in performance between pretrained models and non-pretrained models. Electronic supplementary material The online version of this article (10.1186/s13321-019-0368-1) contains supplementary material, which is available to authorized users.

Introduction
Analysis of compound-protein interactions (CPIs) has become an important prerequisite for both discovering novel drugs for known protein targets and repurposing new targets for current drugs [1][2][3]. Exploring both molecular and proteomic space is a highly challenging and cost-intensive procedure. Each space is enormous and heterogeneous, moreover, most of the CPIs space remains to be discovered. For example, there are roughly 10 8 synthesized compounds potentially developed into novel drugs [4,5] but they are a small fraction of druglike compounds, which the total is estimated on the order of between 10 24 and 10 60 [5,6]. As for the targets of the compounds, there are about 200,000 reviewed human protein records [7]. In vitro experiments are commonly used in identifying CPIs, but it is not feasible to discover molecular and proteomic space only through experimental approaches. In silico models have emerged to aid traditional experiments by narrowing down the search
Traditional in silico models can be grouped into two approaches, which are structure-based methods [12][13][14] and ligand-based methods [15][16][17]. In addition to the conventional approaches, proteochemometrics (PCM) methods have been proposed to predict CPIs by incorporating both ligand and target space within a single model [18][19][20][21]. First, structure-based methods yield reasonable prediction performance and visually interpretable results. Structure-based methods use three-dimensional (3D) simulation for molecular docking to discover CPIs. AutoDock [22], Glide [23], Fred [24], and AtomNet [25] are examples of docking tools. However, the methods have two major limitations: (1) intensive computational complexity and (2) the shortage of 3D structure data for compounds and proteins. Therefore, ligand-based and PCM methods are preferred in most cases.
Secondly, ligand-based methods depend on a basic assumption called the molecular similarity principle [26].
The assumption is that similar compounds are used to interact with similar proteins, where Quantitative Structure-Activity Relationship (QSAR) model is one of the representative examples. With the advent of machine learning (ML) algorithms, ligand-based methods, such as Naïve Bayes (NB) [27,28], random forest (RF) [29], support vector machines (SVM) [30], deep neural networks (DNNs) [31] and multi-task neural networks [32,33], have gained popularity. However, molecular activity alone is not sufficient to identify the bioactivity.
In contrast to ligand-based methods, PCM methods build a model using each compound and protein pair as the input data to fully utilize both proteomic and molecular space. Due to their pair-based modelling, PCM methods are able to predict the interactions between novel compounds and new proteins. PCM methods have recently demonstrated their performance in various tasks such as the identification of new drug combinations [34], prediction of interactions between drug and target [35], and CPIs prediction for G protein coupled receptor (GPCR) and protein kinase targets [36]. In addition, PCM has the potential to utilize information from various multispecies into a single model [18,37]. Therefore, PCM methods have drawn attention in discovering CPI space [20].
DNN algorithms have recently been applied to predict CPI pairs and performed better than other shallow classifiers such as RF, NB and SVM [38,39]. In addition to using basic DNNs called feedforward neural networks (FFNN), a previous study [40] has proposed pairwise input neural networks (PINN). As a variation of a FFNN, a PINN consists of two separated layers and one concatenated layer. Each separated layer is fed with a different feature as the input (i.e. compound and protein) and then each layer is concatenated before classifying the classes. Before the concatenated layers, each separated layer is independently composed without connection from other layers. This architecture allows PINN to reduce the total number of parameters in the networks by about 50% compared to the conventional FFNN without degradation in performance (see Additional file 1: Table S1). The architecture is also suitable for PCM methods, which utilize both compound and protein features. However, the majority of DNNs commonly require a considerable volume of data for each training target. Although the number of public available CPI pairs has grown rapidly, it is still not sufficient to model CPI space [41].
Moreover, as a recent study by Lenselink et al. [38] pointed out, public data can have a large number of errors due to the use of different scientific protocols. The authors presented a high-quality benchmark dataset and compared the performance between various combinations of descriptors, methods (PCM and QSAR), machine learning algorithms, and validation partitioning. The study found that PCM models generally exceed QSAR models under the same conditions. PCM-based DNNs algorithms outperformed the other models on both evaluation sets (temporal validation and random validation).
To complete the modelling of CPI space [20], there is still room for improvement for PCM-based DNNs in terms of representation learning. DNNs can be utilized with three approaches including a classifier, a feature extractor, and an end-to-end learner. As a classifier, DNN algorithms in drug discovery are generally fed with manually crafted features and predict the bioactivity. DNNs can also be used as a feature extractor for compound and protein descriptors [42,43] to fully utilize the information in large-scale dataset such as ZINC for compounds [44] and UniProt for proteins [45]. As an end-to-end learner, DNNs can learn representations from raw data such as SMILES string of compounds and amino acid sequence of proteins. End-to-end learning manages the whole learning process from feature extraction to classification in a single model.
In this paper, we propose a novel multi-channel PCMbased DNN called Multi-channel PINN (MCPINN). In order to make full use of sparse data, MCPINN utilizes three approaches of DNNs which are a classifier, a feature extractor, and an end-to-end learner. This model can be fed with both low and high levels of representations and can incorporate each of them into a single model (Fig. 1). With PINN architecture, this model takes both compounds and proteins into the input layer. It takes SMILES strings, ECFPs and vectors embedded by Mol2vec [42] for compounds and amino acid sequences and vectors embedded by ProtVec [43] for proteins. By incorporating the three approaches of DNNs, MCPINN can learn multiple representations to model the CPI data space. In addition to improving the model, we explore the potential ability of MCPINN to transfer the generalized representations from a high quality and well balanced training dataset to a strongly imbalanced test dataset.
As a proof of concept, we evaluated MCPINN on a standardized benchmark dataset [38] obtained from ChEMBL, using MCC and ROC as evaluation metrics. To investigate the effect of each feature, MCPINN was evaluated with six combinations of single-channel feature pairs. MCPINN was also evaluated with nine combinations of multi-channel feature pairs to explore the synergy effects of low and high levels of representations. The models were investigated in terms of not only highest performance but also initial performance and convergence speed. To test whether MCPINN can transfer general representations of compounds and proteins to a new task, we pretrained models on a training task, which is the benchmark dataset used above, and then finetuned the pretrained models on a test task Tox21 [46]. The transferability of MCPINN was evaluated in terms of initial performance, speed of convergence, and highest performance using two metrics for validation MCC and PRC. Therefore, this study contributes to "the complete modelling of CPI space" [20] by full use of representation ability of DNNs as a classifier, a feature extractor, and an end-to-end learner and additionally by transferring the generalized representations from training tasks to test task.

Results and discussion
Investigating the representation learning ability The first part of this study focuses on the representation learning ability of MCPINN. To figure out the contribution of each feature on the predictive performance of the model, MCPINN was evaluated with fifteen combinations of feature pairs, which contained six pairs from single-channel features and nine pairs from multi-channel features. There are three features for compounds: SMILES, ECFP, and Mol2vec, where SMILES is a lowlevel representation. The performance based on the feature concatenated ECFP and Mol2vec was evaluated but omitted because the concatenated feature models did not provide improvement in performance compared to the Mol2vec or ECFP models separately (as can be seen in Additional file 1: Table S2). Two features are used for proteins: ProtVec and the amino acid sequence which is a low-level representation. For low-level representations (SMILES and amino acid sequence) Dilated CNN is applied as an end-to-end learner. Recurrent Neural Network models were also trained but omitted due to their poor performance, which can be seen in Additional file 1: Figures S1 and S2.
There are fifteen models based on the combinations of feature pairs and are listed in Table 1 with shortened names. The models were evaluated on a benchmark dataset [38] using two metrics for validation the Matthew Correlation Coefficient (MCC) and Receiver Operating Characteristic Area Under the Curve (ROC).

Comparison between single-channel models
Above all, to investigate the effect of each feature on the predictive performance of the model, the six feature pairs are explored with Single-channel PINN (SCPINN). SCPINN is a basic model that is fed with only one feature for each protein and compound respectively as shown in Table 1. The prediction performance of each model is shown in Fig. 2. The average performance of SCPINN models was an MCC of 0.636 ± 0.03 and a ROC of 0.892 ± 0.02. Overall the differences in performances between the metrics scores were similar to each other. It is observed that the biggest difference in performance between the SCPINN models was the use of a high-level representation (ECFP and Mol2vec) in the chemical feature instead of a low-level representation (SMILES). For example, the average performance of the models using ECFP and Mol2vec for compounds was an MCC of 0.66 ± 0.008 and a ROC of 0.90 ± 0.004, while the average performance of the models using SMILES was an MCC of 0.60 ± 0.014 and a ROC of 0.87 ± 0.007.
On the other hand, the models using ProtVec did not outperform the models using amino acid sequence with Dilated CNN for the overall models, regardless of the types of chemical features used. The average MCC of models using amino acid sequence was 0.646 (± 0.023) and mean ROC was 0.896 (± 0.011), while the average MCC of models using ProtVec was 0.627 (± 0.029) and the mean ROC was 0.887 (± 0.015).
This difference in performance seems to be based on whether or not the feature extraction method is able to capture the order of the amino acid sequences in the feature vector, in addition to the content itself. The Dilated CNN model can featurize the entire sequence of a protein in terms of the order and content, whereas ProtVec has a limitation in that it does not reflect the order of the sequence in the feature vector. ProtVec divides the sequence into N-grams to make the word units, performs individual embedding on each N-gram word, and then sums up all the embedding vectors regardless of the orders. Therefore, different proteins could have the same embedding vectors with ProtVec, provided the same N-grams are used. Secondly, there is room to improve operations in Mol2vec and ProtVec to prevent incorrect representations of embedded compounds and proteins. In Mol2vec and ProtVec, the sum operation reconstructs embedded word vectors (i.e. Morgan substructure or N-gram amino acid sequence) into a sentence vector (i.e. compound or protein). Since the number of sum operations is dependent on the number of words in the sentence, applying these operations can significantly alter the embedded value of the sentence, regardless of the actual meaning of the sentence. To prevent information distortion, the sum operation in Mol2vec and ProtVec should be improved. Therefore, in the next section, we first refine the sum operation.

Improving Mol2vec and ProtVec
To refine the sum operation in Mol2vec and ProtVec, we tested two types of weighted average operations, which are arithmetic mean and Term Frequency Inverse Document Frequency (TF-IDF) [47]. The former sets the weight of each word according to the length of the sentence while the latter sets the weight of each word by TF-IDF (see "Methods and materials" section). Table 2 shows the predictive performance of nine combinations of feature pairs using the original methods and the proposed methods for both Mol2vec and ProtVec. The proposed methods performed better than original methods with the exception of the arithmetic mean method on proteins. The best method for Mol2vec is the arithmetic mean method having an average MCC of 0.659 ± 0.013 and an average ROC 0.906 ± 0.004, compared to TF-IDF weighted average method (MCC of 0.657 ± 0.013 and ROC of 0.903 ± 0.002), and the original methods (MCC of 0.649 ± 0.014 and ROC of 0.903 ± 0.006). For ProtVec, the TF-IDF weighted average method outperformed the other models with an average MCC of 0.673 ± 0.04 and an average ROC of 0.909 ± 0.003. Among all of the usage of Mol2vec and ProtVec, the best performing feature pair is arithmetic mean method for the former and TF-IDF weighted average method for the latter, where the usage of the pair showed an MCC of 0.678 ± 0.002 and a ROC of 0.912 ± 0.002.
It is observed that these improved methods can more accurately capture the contents of each compound and protein. The sentences (i.e. compounds and proteins) within each document (i.e. bioactivity dataset) have specific contexts and characteristics, which the entire corpus set (i.e. ZINC and UniProt) cannot represent. In particular, TF-IDF assigns a weight to each word in a sentence, so that TF-IDF weighted average method is able to more finely capture the characteristics and contexts inherent in the document.
In the case of Mol2vec, the TF-IDF weighted average method has a slightly lower performance than the arithmetic mean method. It seems that the TF-IDF weights from a specific document can be used to bias the information toward the document and reduce the generalization performance. In summary, all words were first embedded within the whole corpus, and then sentences were represented by weighting each word through a document. As a result, SC 4 performed better than original one, where MCC increased to 0.678 from 0.642 and ROC increased to 0.912 from 0.900.

Comparing the performance of multi-channel models
To figure out the synergy effects of a combination of both low and high level representation, the nine MCPINN models based on multi-channel feature pairs are evaluated as shown in Table 1. In order to improve the readability of this paper, the three multi-channel features are abbreviated as follows: ProtVec with amino acid sequences is ProtVec AA , Mol2vec with SMILES strings is Mol2vec SS , ECFP with SMILES strings is ECFP SS . It is observed that the effect of multi-channel was different between proteins and compounds, as shown in Fig. 3. In the case of protein features, it was observed that the usage of ProtVec AA performed better than the others (average MCC of 0.658 ± 0.03 vs. 0.649 ± 0.03 and average ROC of 0.902 ± 0.02 vs. 0.897 ± 0.02). End to end learning channel with Dilated CNN seems to mainly represent the order (sentence level) of the amino acid sequence, while ProtVec channel represents the importance (word level) of each amino acid sequence in the protein. This suggests that the proposed multi-channel architecture can utilize both channels to capture features from both sentence and word perspectives for proteins.
Contrary to expectations, multi-channel models for compounds demonstrated very different results between In addition, the usage of Mol2vec SS also resulted in lower training performance than ECFP SS , where the average training performance was an MCC of 0.97 ± 0.006 for the models using ECFP SS and an MCC of 0.95 ± 0.007 for the models using Mol2vec SS . Therefore, a careful selection of representations is required to achieve better performance. These results suggest that the richness of the features of compounds highly depend on the base representations. For example, compounds are represented in the form of a two-dimensional graph as the raw data for ECFP and Mol2vec, where they divide the graph into substructures and define each part to extract compound features. In contrast, Dilated CNN extracts features from an onedimensional SMILES strings and it seems to capture less generalized representations from the strings compared to the representations from ECFP and Mol2vec. In this study, sequence data was used for the multi-channel architecture but there are a variety of other data types that can be embedded and used for a new channel. Therefore, the more embedding methods applied to a variety of data types such as graphs [48], heterogeneous networks [49], and nodes [50], the more biological and molecular information (i.e. pathway and drug-drug interactions) can be fully utilized for drug discovery, poly-pharmacology, side-effect prediction, and drug resistance.

Ranking the features and models
We compared and ranked fifteen models including the six SCPINN models and the nine MCPINN models introduced above. To compare between the models, we calculated two z-scores for each model and metric (MCC and ROC) and averaged them as shown in Fig. 4 and Table 3.
To verify the validity of the difference between the z-scores, the following statistical tests were performed: the paired Student's t Test and the F Test.
Among the chemical features, the usage of Mol2Vec showed the best performance with an average z-score of 0.94 ± 0.01 , compared to ECFP SS ( 0.57 ± 0.02 ), ECFP ( 0.53 ± 0.02 ), Mol2vec SS ( − 0.25 ± 0.06 ), and SMILES ( − 1.79 ± 0.02 ). For the Student's t test, the usage of Mol2vec and SMILES are shown to significantly differ from all other features with a p value < 0.05 . Likewise, the usage of ECFP SS and ECFP differs significantly from all features with a p value < 0.05 with the exception of the usage of Mol2vec SS , where the p value is 0.06 and 0.07 respectively (Additional file 1: Table S3). For the F Test, the differences in variances are also noticeable from all features with a p value < 0.05, with the exception of ECFP and ECFP SS , where the p value is 0.38 for each other (Additional file 1: Table S4). Therefore, Mol2vec and Mol2vec SS showed significant differences in performance of both mean and variance, while ECFP and ECFP SS showed significant differences in mean performance.
Among the protein features, the usage of ProtVec AA outperformed the other features with an average z-scores of 0.21 ( ± 0.009 ), compared to ProtVec ( − 0.14 ± 0.008 ) and AA sequence ( − 0.08 ± 0.001 ). It is observed that the usage of ProtVec AA performed better than the others in terms of means and variances with a p value < 0.05, while ProtVec and AA sequence did not differ significantly (p value is 0.21 and 0.06 for the means and variances respectively (Additional file 1: Tables S5, S6). It is observed that there are considerable synergy effects of multi-channel for proteins. Therefore, these statistical results indicate that the usage of Mol2vec and ProtVec AA outperformed the usage of the other features.
The best model was MC 2 with a z-score of 1.22 ( ± 0.001 ), followed by SC 4 with a z-score of 0.93 ( ± 0.020 ), and MC 9 with a z-score of 0.75 (± 0.017). It is observed that there were significant differences between the highest model and the lowest model compared to the other models (as can be seen in Additional file 1: Tables  S7, S8). For example, for the Student's t test MC 2 and SC 2 were shown to significantly differ from all other models with a p value < 0.05. Likewise in variance MC 2 and SC 2 were significantly different from the other models with the p value < 0.05. So far we have only looked into the highest performance. In the next section we look further into initial performance and the speed of convergence.

Comparing convergence speed
In addition to the maximum performance, also noticeable are the differences in initial performance and convergence speed between SCPINN and MCPINN. Initial performance was measured by the performance at the first epoch and the speed of convergence was measured by the actual run time at 98% of the highest performance of the model. In order to compare the convergence speed of each model more precisely, we mainly measured actual run time and secondarily labeled the number of epochs. For more information about convergence speed against training epochs, refer to the Additional file 1: Figure  S3. The machine specifications for the experiments are described in "Methods and materials" section-Hardware used. There are more thresholds that were tested for convergence performance such as 95% , 98% , and 99% in Additional file 1: Table S9. The top 3 performing models ( MC 2 , SC 4 , and MC 9 ) and baseline model ( SC 1 ) were compared. Each model showed differences in the number of parameters, training time on an epoch, and the performance but there appears to be no direct correlations  between them (as can be seen Additional file 1: Figures S4 and S5). It is observed that MCPINN performed better than SCPINN in terms of initial performance and convergence speed as shown in Fig. 5. Initial performance was an MCC of 0.47 ± 0.004 for MC 9 , 0.43 ± 0.005 for MC 2 , 0.40 ± 0.003 for SC 1 , and 0.38 ± 0.016 for SC 4 . The time it took to reach 98% of the highest performance was 11 min (18 epochs) for MC 9 , 41 min (113 epochs) for MC 2 , 50 min (102 epochs) for SC 1 , and 55 min (201 epochs) for SC 4 . SC 4 and MC 9 showed the most contrasting differences in the convergence speed and the highest performance. Even though the former performed a little better than the latter in performance with an MCC of 0.678 versus 0.674, it took 104 min to outperform the latter.
While the exact cause in these differences cannot be proven, it seems that low-level representations from Dilated CNNs contributed to a non-negligible portion in these differences between the models. Because it is worthwhile examining these differences between the models, let us discuss these phenomena in the perspective of the information bottleneck (IB) theory of deep learning [51]. The authors claim that "DNNs undergo two distinct phases which consist of an initial fitting/ memorizing phase and a subsequent compression/forgetting phase, which is related to the high generalization performance of DNNs" [51]. In this point of view, following explanations can help account for differences in convergence speed: (1) multi-channel architecture can help to construct better representations and reduce the length of two phases because there is little information to be compressed or forgotten. (2) single-channel architecture generally need more training to discover appropriate representations for both fitting phase and compression phase, because there are not enough features. In summary, multi-channel architecture can improve convergence speed as well as the performance.

Exploring the potential of transfer learning
While this study has focused on representation ability of MCPINN in terms of the performance and convergence speed, this section further explores the transferable ability of MCPINN to generalize representations from training tasks to related testing tasks. To test whether MCPINN can capture general information for compounds and proteins, we pretrained MC 2 on the benchmark dataset and finetuned the model on Tox21 dataset [46]. In order to improve the readability of this section, the pretrained models are abbreviated as follows: PM i , where i is the number of epochs pretrained on training task, so non-pretrained model is PM 0 . PM i was finetuned on the Tox21 training set with early stopping on the validation set and evaluated on the test set, where the Tox21 dataset was split as suggested by DeepChem [52].
It should be noted that the two datasets are distinct, where the benchmark dataset is based on biophysics, while the Tox21 dataset is based on physiology [53]. The benchmark dataset, obtained from ChEMBL [41], focused on bioactivity of small molecules, while Tox21 measured toxicity results in nuclear receptor and stress response pathways in human body. Because Tox21 dataset is strongly imbalanced dataset with the percentage of positives being 7.49% (5957 positives from 79,585 all data points), the performance of models was measured using MCC and Precision-Recall AUC (PRC) instead of ROC, where PRC can provide more accurate prediction when applied to imbalanced classification scenarios [54].
To investigate the potential of transferability of MCPINN, we have compared the performance of the models pretrained in different epochs in terms of highest performance, initial performance, and convergence speed [55,56]. First, pretrained models performed higher than non-pretrained model. The non-pretrained model PM 0 had an MCC of 0.43 and a PRC of 0.48 as shown in Fig. 6. The pretrained models from PM 30 to PM 110 outperformed PM 0 with a paired t test p value < 0.05 for both MCC and PRC with an exception of PM 85 , where the p value was 0.053 (Additional file 1: Table S10). It is observed that the overall performance of the models rose up to PM 55 and then declined, where it seems that the decline is because of overfitting on the training task dataset.
In contrast, there were small differences in initial performance and convergence speed between the models. We looked into the finetuning phase of the three models including PM 0 , PM 55 , and PM 135 , in order to investigate the generalization performance according to the  Table 4 and Fig. 7, PM 0 performed slightly better than other models until finetuning epoch 10, but the performance became lower than other models as finetuning continued. For example, initial performance was an MCC of 0.16 ± 0.03 for PM 0 , 0.11 ± 0.02 for PM 55 , and 0.08 ± 0.03 for PM 135 . After finetuning epoch 11, PM 55 started to outperform PM 0 and PM 135 did so after finetuning epoch 40. In addition to initial performance, it is observed that there were similar performance in convergence speed between models. The number of finetuning epochs to reach 95% of the highest performance was 46 finetuning epochs for PM 55 , 56 finetuning epochs for PM 135 , and 60 finetuning epochs for PM 0 .
From the results we can see there is still room for improvement. The aim of transfer learning based on PCM methods is high performance with minimum finetuning. Due to the flexibility of PCM method, MCPINN can predict any CPI pairs, while the performance without finetuning was poor as can be seen in initial performance of each model. Since there are still a lot of small molecules with only a small amount of bioactivity data, further study of transferring general information covering CPIs space is required.

Conclusions
In this paper we proposed a novel multi-channel PINN (MCPINN) based on PCM methods to fully utilize CPI data. MCPINN utilizes three approaches of DNNs which are a classifier, a feature extractor, and an end-to-end learner to maximize the representation learning ability. We evaluated full combinations of feature pairs to investigate the effects of each pair. We also compared SCPINN and MCPINN in terms of initial performance and the speed of convergence. In addition to improving the models within a high quality and well balanced dataset, we explored the transferable ability of MCPINN to generalize representations from training tasks to related testing tasks, which consist of a strongly imbalanced dataset. To the best of our knowledge, MCPINN is the first method to incorporate low and high level representations in a single model.
As discussed above, our results lead to a number of conclusions. For Mol2vec and ProtVec, we suggested that a weighted average operation is a better alternative to the sum operation in representing compounds and proteins. MCPINN using the feature pair of ProtVec AA and Mol2vec outperformed all other models with statistically significant differences. The usage of ProtVec AA performed better than others. It suggested that a multichannel architecture can utilize both channels to capture the order and the content of amino acid sequences. The usage of Mol2vec showed statistically significant differences from the other features for compounds. In particular, the multi-channel models using Mol2vec SS performed worse than the single-channel models using Mol2vec separately. Depending on the combination of compound and protein features, multi-channel models did not guarantee better performance than single-channel models,   so a careful selection of representations is required to achieve better performance. The multi-channel architecture can improve initial performance and convergence speed. It seems that the architecture can help to construct better representations and reduce the length of training phase based on memorizing phase and forgetting phase in terms of IB theory of deep learning. Additionally, we explored the potential of transferability of MCPINN in terms of initial performance, speed of convergence, and highest performance. Pretraining on training task improved highest performance, while it did not improve convergence speed and initial performance. It seems that there is room for improvement to transfer the generalized representations from training tasks to test task.
In conclusion, MCPINN can improve the representations in terms of initial performance, convergence speed, and highest performance. Moreover, we expect that more biological and molecular information can be utilized as a part of multi-channel for various tasks such as drug discovery, poly-pharmacology, side-effect prediction, and drug resistance.

Methods and materials
Datasets A high quality dataset [38] was employed as a benchmark dataset for the training task. Fifteen models are evaluated on the benchmark dataset. The dataset covers 0.13% of the total available bioactivity matrix space in ChEMBL, where there are 314,767 observations from 250,412,295 possible data points produced by 204,085 compounds and 1227 protein targets. Percentage of the positives in the dataset is 54.7%. We used Tox21 dataset for the test task in transfer learning. Tox21 dataset has been used in the 2014 Tox21 Data Challenge, where there are 79,585 measurements for 8014 compounds on 12 different targets. Tox21 dataset is strongly imbalanced and the percentage of positives is 7.49%.
DNNs try to minimize differences in the distribution of data between the prediction and target due to the usage of cross entropy as loss function. Therefore, training models on imbalanced data is a challenge. One of the basic solutions is to set higher weights on the positives than the negatives. In addition, it is also difficult to appropriately split the dataset into a training set, a validation set, and a test set. Therefore, we used the data splitting method and the weight value for the positive classes as suggested by Deepchem [52].

High level representation descriptors
We used Extended-Connectivity Fingerprints with diameter of 4 (ECFP4), Mol2vec [42], and ProtVec [43] to get high level representations. ECFPs is one of the most popular representation in cheminformatics and ECFP4 have shown promising performance among various fingerprints [57]. RDkit [58] was used for ECFP4 with 1024 dimensional binary vector.
ProtVec and Mol2vec are unsupervised machine learning approaches for embedding proteins and compounds. These algorithm are inspired by a technique called Word-2Vec [59] in Natural Language Processing (NLP). As a metaphor by NLP, molecules and proteins are considered as sentences. Morgan substructures and N-gram amino acid sequences are considered as "words", and large-scale databases such as ZINC [44], ChEMBL [41] and UniProt [45] are considered as large corpus datasets.
For the protein embedding model, we train the embedding model on protein sets obtained from UniProt (release 2017_09) and ChEMBL (version 23). All duplicate sequence and same protein id with various amino sequences are removed. The number of sequences for training embedding model is 553,195. The embedding model is based on Skip-gram model and the model is trained with following hyperparameters: dimension of the representation is 300, window size is 35, and minimum count is 2. In Tox21, the target "SR-MMP" has no amino acid sequences, so the embedding value is zeros. Molecular corpus dataset, obtained from ZINC and ChEMBL (version 23), contains about 19.9 million compounds using the approach suggested in [42], and we used a pretrained embedding model the authors proposed.
Embedded sentences are composed of the group of embedded words. We build two types of weighted average methods, which are arithmetic mean and TF-IDF weighted average method, to refine the original sum method. The original sum method is as followed: where S sum is a embedded sentence produced by the method, N is the number of words in the sentence, and w i is a ith embedded word in the sentence. However, the number of sum operations is dependent on N of each sentence, so it can alter the embedded sentence, regardless of the actual meaning of the sentence. Instead, arithmetic mean method is as followed: where S mean is a embedded sentence produced by the method. This method divides each word by the length of the sentence. Therefore, the same word can have different embedded value in each sentence due to the differences in length. Moreover, TF-IDF [47] weighted average method is as followed: where S tf −idf is a embedded sentence produced by the method and t w stands for TF-IDF weight value for a word w. This method sets the importance of each word by TF-IDF, so the same word has same embedded value in every sentence. To calculate weight value of TF-IDF, scikitlearn (version 0.19) is used based on compounds and proteins in benchmark dataset and Tox21.

Low level representation descriptors
We used low-level representation data for end-to-end learning models, where they are amino acid sequences for proteins and SMILES strings for compounds. Both sequences were tokenized and then encoded into onehot binary vector with fixed length. Tokenizing process produced 24 single characters from the proteins and 57 single characters for SMILES as suggested in [60], where the characters are extracted from benchmark dataset and the Tox21 dataset. The tokenized strings were converted into one-hot encoded representations, which assign the corresponding single token to one and the others to zero. In order to use the sequences as an input for the machine learning model, we set the sequences to a fixed length with post truncation or zero-padding. If the sequences are longer than the fixed length, they are trimmed by removing from the end of sequences to the fixed length, unless they are filled with zero from the end of the sequences to the fixed length. In determining the fixed length of sequences, there is a trade-off between information preservation and computational efficiency. We chose the fixed length 100 for compounds and 700 for proteins, where a percentile of 75% for SMILES strings is 63.0 and a percentile of 75% for amino acid sequences is 712.2 as shown in Figs. 8 and 9.

Transfer learning
Transfer learning focuses on whether machine learning model can transfer generalized representations from training tasks to a different but related test tasks. While there are several factors that affect finetuning methods, two important factors are generally considered [55,56]. The factors are the size of the test task's dataset (i.e. small or large) and similarity of test task (i.e. the content of data or classes and balance of data or classes). There are four basic strategies to finetune the pretrained models on test t w w i tasks as followed: (1) If the dataset of test task is large and the task is very similar to training task, finetuning full networks is suggested since the risk of overfitting is low. This case is expected to demonstrate promising performance. (2) If the dataset of test task is large and the task is very different from the training task, there are two options which are finetuning full networks or not pretraining the model on the training dataset. In practice, it is suggested to finetune the full networks to reduce training time. (3) If the dataset of test task is small and the task is very similar to the training task, finetuning full networks is not suggested due to the risk of overfitting. Instead, it is suggested to finetune the simple classifier to avoid overfitting. (4) If the dataset of test task is small and the task is very different from the training task, a simple classifier is not suggested due to the differences between tasks. It is suggested to initialize the top layers and freeze the other layers to finetune the layers and classifier, since the top layers contain more task-specific representations.
The benchmark dataset is four times larger than Tox21 dataset and the tasks are different in terms of biophysics versus physiology as categorized in [53]. Moreover, the benchmark dataset is balanced, while Tox21 dataset

DNNs
All DNN models are created using Keras [61] and Tensorflow [62]. We tested various architectures, parameters, and hyperparameters to optimize DNN models in initial cross-validation phase as shown in Table 5. All tested models were validated with five-fold cross-validation on validation data (20% of training data) for 400 epochs with early stopping in order to find the optimal network configuration. Then the optimal model was trained on the full training data and evaluated on test data.
Adam is generally used in DNNs due to efficient and fast training performance because the step size is not affected by the value of the gradient. We used the hyperparameters 0.9 for β 1 and 0.999 for β 2 as suggested [63]. However, learning rate was 0.0005 and weight decay was not zero to achieve more stable training, where weight decay reduces the learning rate over each update. Since the benchmark dataset is very sparse, small batch size can mislead the training model to local optimum. Therefore, we set mini-batch size 1024 for the generalized performance as suggested [64]. All weights and biases were initialized from a uniform distribution within [− 3/fan in , 3/fan in ] , where fan in is the number of input units in the weights, which is called Lecun uniform distribution. Lecun uniform distribution performed better than random uniform distribution and truncated normal distribution in terms of performance and convergence speed, because it leads to efficient backpropagation calculations [65].
Rectified linear (ReLU) units are commonly used in DNNs because they do not suffer from vanishing gradient and their training speed is fast. However, ReLU units ignore the negative values, so there is information loss called "dying ReLU" [66]. Exponential linear units (ELU) [67] was introduced to solve the problem. ELU and ReLU are in identity function form for non-negative inputs, but for negative inputs, they are different, where if x < 0 , f (x) = α(e x − 1) for ELU and f (x) = 0 for ReLU. ELU can capture information in the negative value. Therefore, we used following parameters for the final DNN models: (1) as an optimizer Adam with 0.9 beta 1 and 0.999 beta 2 is used as suggested [63], (2) learning rate is 0.0005, (3) number of epochs is 500, (4) mini-batch size is 1024, (5) Lecun uniform distribution, (6) the weight decay is 0.00001, (7) activation function is ELU.

DNNs: end-to-end learning
We built three types of end-to-end DNNs based on convolution neural networks (CNN) and recurrent neural networks (RNN). RNN is designed to learn sequential data and CNN has multiple filters which are incorporated with each other to discover various representations. These model have shown promising performance for sequential data in various domains. Among RNN models, long short-term memory (LSTM) [68] and bidirectional LSTM (BLSTM) [69] have outperformed conventional We used dilated convolution neural networks (Dilated CNN) [71] among end-to-end learners. Dilated CNN is a convolution neural networks with skip layers. Conventional CNN learns long-term dependency by reducing the size of the data, but it results in information loss. In contrast, Dilated CNN can learn long-term dependency efficiently with skip layers. The layers have wider receptive fields compared to conventional layers. The size of kernel is the length of the convolution window and it affects the long-term dependency of given sequences. The basic suggestion is a small kernel size (i.e. 3) to achieve efficient training and less number of parameters [72]. However, we chose larger size of kernel, since Pro-tVec and Mol2vec already captured the features in terms of local perspective.
The number of filters determines depth of the output volume called feature map, which is the result of the convolution layer. If the number of filters is too large the model can suffer from overfitting, otherwise the model can suffer from underfitting. In computer vision domain, the number of filter is large but we tested smaller numbers of filter due to the sparseness of CPI data space. The embedding layer is the first layer for one-hot encoded vectors. The dimension size 32 and 16 was tested, but there were little differences in performance. Therefore, the final value of network architecture and hyperparameters for Dilated CNNs were (1) the number of filters is 16, (2) the kernel size is 12, (3) an embedding layer with 16 dimension is used to reduce the number of parameters, and (4) valid padding to reduce the shape of the feature maps in each layers.
For LSTM and BLSTM, the final value of network architecture and hyperparameters were: (1) units are 256, which is the dimensionality of output, (2) set forget bias as suggested [73].

DNNs: pairwise input neural networks
Pairwise input neural network (PINN) is used for MCPINN and SCPINN. PINN is a variation of feedforward neural networks and is a more suitable architecture for PCM methods. It consists of separated layers with two input and concatenated layers. For MCPINN, all channels are merged in the concatenated layer as shown in Fig. 1. Since the separated layers are independently composed without connection from other layers, each input channel layers build representations independently for each input. Moreover, the architecture can balance the ratio of each feature by controlling the number of nodes in the last separated layers. For example, although the input dimension of ECFP is 1024 and the input dimension of ProtVec is 300, the dimension of each representation is the number of nodes in the last separated layers. In contrast, DNNs can be biased to the feature of larger input dimensions.
We normalized the high-level representations with zero mean and unit variance to achieve stable training, because outliers can degrade the performance of machine learning algorithms in terms of prediction, learning speed, and the convergence of the algorithms. In particular, many gradient based algorithms (i.e. deep learning) are often designed with the assumption that input data is nearly standardized data, which is generally obtained by subtracting the mean and scaling the data to unit variance. We tested separated layers from 1 to 4 and concatenated layer from 1 to 2. The number of concatenated layer is 1 for both SCPINN and MCPINN, but the number of each separated layers is 2 for SCPINN and 1 for MCPINN to reduce overfitting, where the parameters of end-to-end channel was added in MCPINN. To prevent overfitting, we used 10% dropout on initial layer and 50% on hidden layers and early stopping.
In our experiments, the final value of network architecture and hyperparameters for PINN were: (1) the number of each separated layers is 2 for SCPINN and 1 for MCPINN, (2) the number of each concatenated layer is 1, (3) the number of units in each separated layer is 1024 and 256, (4) the number of units in each concatenated layer is 256, (5) dropout rate is 0.5 (6) each features are normalized with zero mean and unit variance.

Performance metrics
For the performance evaluation, we used three metrics, which are Matthew Correlation Coefficient (MCC), Receiver Operating Characteristic Area Under the Curve (ROC), and Precision-Recall Area Under the Curve (PRC). Above metrics are commonly used in binary classification to evaluate the quality of the performance. ROC space is determined by the false positive rate (FPR) versus true positive rate (TPR) called recall as x and y axis, where FPR and TPR is calculated by following formula: TPR = TP/(TP + FN ) and FPR = FP/(FP + TN ) , where TP is the number of true positives, FN the number of false negatives, FP the number of false positives, and TN the number of true negatives. It means ROC shows relative trade-offs between true positive and false positive. The value of ROC is between 0 and + 1, where + 1 indicated perfect prediction, 0.5 means random prediction, and 0 indicates totally wrong prediction.
PRC can provide more accurate prediction when applied to imbalanced classification scenario than ROC, because PRC put more importance on the TPR in case of imbalanced dataset. ROC and PRC share TPR (recall) on same axis, but PRC uses precision for the other axis, where precision is calculated by following formula: precision = TP/(FP + TP) . MCC is generally regarded as being one of the best metrics because MCC is more useful than other metrics when the two classes are very different. MCC is calculated by following formula: The value of MCC is between − 1 and + 1, where + 1 indicates perfect prediction, 0 means random prediction, and − 1 represents totally wrong prediction.

Software used
Python (version 2.7) was used with the following libraries: Keras (version 2.2.0) and Tensorflow (1.9.0) for the neural networks, RDKit (version 2017.03.3) for the calculation of the fingerprints and descriptors, scikit-learn (version 0.19) for splitting validation, normalization and performance evaluation, SciPy (version 1.2.0) for statistical analysis including students t test and Fisher F test, ProtVec for the protein descriptors, and Mol2vec for the molecule descriptors.

Hardware used
A Linux server running Ubuntu 16.04 was established for experiments. The server was equipped with a Xeon E5-2620 v4 processor, 96 GB RAM, and four NVIDIA GeForce GTX 1080 Ti for Graphics Processing Units.