Application of Artificial Intelligence Technology in Medical Imaging

Based on massive medical image data, through reasonable system design and parameter optimization, the AI-assisted diagnosis system can help doctors make more clinical decisions, reduce their daily work pressure, reduce unnecessary invasive tests, and thus improve the quality of life of patients. This paper mainly studies the application of artificial intelligence technology in the field of medical imaging. In this paper, deeplesion public data set is used as the training set to study and optimize the convolutional neural network structure and parameters of deep learning algorithm based on the basic convolutional neural network model, aiming at the pathological features of lung cancer in the data set, such as lobulation sign and burr sign. At the same time, combining with the characteristics of neural network recognition and data set, the data set is preprocessed to highlight the features of the image and improve the accuracy of deep learning algorithm. Moreover, an auxiliary recognition system is developed around the optimized algorithm, so that the deep learning algorithm can intuitively show the automatic recognition of lung cancer through the recognition system after a large amount of data training.

Introduction
Artificial intelligence has now penetrated into our lives, from personal assistants, security and autonomous driving to healthcare, e-commerce, finance and education, AI is closely related to our lives. Artificial intelligence can realize automatic recognition and diagnosis function in the field of intelligent medical treatment. For example, as long as the image of the mole on the arm is obtained, the computer can clearly determine whether it is a mole or skin cancer, and a group of eyeball images can be taken to determine whether there is a risk of cataract or deepened myopia [1]. The development of artificial intelligence has made people's imagination unrestrained. On the one hand, medicine is slowly being miniaturized from large equipment in hospitals through intelligence, such as the skin cancer test mentioned above, where people collect images from their mobile phones and upload them to a cloud server so that a computer can make a diagnosis. On the other hand, the smarter a medical device is, the more complementary it is to a doctor. Intelligentization of medical treatment brings convenient and quick diagnostic assistance to doctors, reduces the working pressure of doctors, and brings efficient medical treatment and low price to patients [2]. In the face of complex diseases, experienced doctors will inevitably misdiagnose, while the new technology of intelligent medical treatment can extract accurate and effective information from complex diseases to assist doctors in efficient diagnosis.
Intelligent medicine is one of the most popular research directions in the medical field in recent years. Most of the algorithms adopt deep learning method. Deep learning algorithms have achieved good results in natural images, and deep learning algorithms have been gradually applied to various fields, including data mining, traffic services and medical diagnosis. In the field of medical diagnosis, Stanford University realized the recognition of skin cancer through deep learning algorithm, trained the CNN network through 129,450 clinical data, and the accuracy rate was very close to that of doctors [3]. In one study, the deep learning algorithm InceptionV3 was used to train 9649 IHC staining and 2490 H&E staining lung cancer tissue sections, and the accuracy of AI recognition of lung cancer reached 99% [4].
In most of the open deep learning algorithms, lung cancer deep learning algorithms are relatively few or not fully open compared with other fields. However, lung cancer images are quite different from other cancer images, which requires some optimization based on the algorithm. At the same time, the market lung cancer diagnosis system is expensive and requires a large number of comprehensive medical data, still many hospitals can not afford. Therefore, the development of medical imaging detection of lung cancer based on artificial intelligence technology has great theoretical significance and practicability.

Overview of Convolutional Neural Networks
Convolutional Neural Network (CNN) is a major research direction of deep learning. It is the first successfully trained neural network containing multi-layer structure, widely used in the processing of one-dimensional sound signals, two-dimensional images, three-dimensional image sequences and video signals, especially excellent in the processing of two-dimensional images. In recent years, in international conference papers such as ICCV, CVPR, NIPS and ECCV, as well as in ILSVRC competition, solutions to computer vision problems have been involved, and almost all models have been completed based on CNN [5][6].
The structure of CNN can be understood as a multi-layer feedforward network, each layer is composed of multiple two-dimensional planes, each plane is composed of multiple neurons, and its basic structure consists of four parts: Input layer, convolutionallayer, poolinglayer and outputlayer [7].
The input layer can directly receive two-dimensional visual patterns, multi-channel images and image sequences without manual selection or design of appropriate features as input.
In CNN, the convolution of the convolution layer is general image convolution, namely twodimensional discrete convolution. The formula of image convolution is as follows: Where I = F(x, y) represents an image, F(x, y) is the gray value of the pixel in row x and column y of the image, and W(u, v) is the gray value of the pixel in row u and column v of the convolution kernel.
What is special about convolution in CNN is that its convolution can be trained. The down-sampling layer, also known as the pooling layer, is generally alternately connected with the convolutional layer. The core of the down-sampling layer is to conduct sub-sampling of images, so as to reduce the amount of calculation and retain key information [8]. In CNN, the convolution layer, pooling layer and full connection layer usually introduce nonlinear factors through activation function, so that the model can better solve complex nonlinear problems [9]. Common activation functions include Sigmoid, Tanh and ReLU.
The Sigmoid function is a monotone and continuous function, and its expression is shown in Equation (2). The input can be mapped to (0,1) and the derivative is easy to be obtained. However, the soft saturation of the function itself makes it easy for the gradient to disappear in the training process.
The tanh function expression is shown in Equation (3), which is a zero-centered output function. Although the convergence rate is improved compared with Sigmoid function, it does not solve the problem of gradient disappearance in the training process.
Relu is a linear unsaturated function. Compared with the previous two functions, its biggest advantage is that the computational complexity is reduced and the optimization speed is very fast. On the one hand, the unsaturation of ReLU improves the sparsity of the network, but once suppressed (zero) neurons may not be activated again [10]. Different activation functions have their advantages and disadvantages, and scholars have been working on the research of activation functions with better performance.

Image Processing
The spectrum map is intercepted according to the coordinates of the region where the spectrum map is located provided in the Deeplesion file, and the region information in the Deeplesion file is obtained by referring to the relevant information of the US region calibration module in the information module specification published by Deeplesion.
Because the coordinate Settings of the enrolled spectrum are inconsistent, the peak information in the spectrum waveform cannot be reflected in the spectrum image. In order to avoid the loss of the peak information, we need to normalize the spectrum first. In the normalized spectral image, the peak information of the waveform can be reflected by the height of its highest point. Specific operation: look for the minimum scale of spectrum in DICOM data set (the minimum interval in X direction is 0.004s, and the minimum interval in Y direction is 0.078cm/s), and normalize all spectrum graphs under this scale.
Normalized spectral images contain different number of signal cycles and different image sizes, so we standardized the images. Specific operation: First, the signal of two cycles is extracted from the spectrum diagram for analysis. Secondly, we construct the background image with the maximum image size (1700x1700, all pixel values are 0), and place all normalized spectral images in the center position of the background image with this size.
In order to further reduce the amount of image data, simplify the image, and highlight the spectral waveform contour, we normalized the unified size of spectral images for image binarization.
Binary image plays an important role in image classification, image binarization of the original image can be turned into black and white images, simplify the image data at the same time highlight the outline of main goal, on the one hand able to more clearly reflect the characteristics of the local and overall, on the other hand, the amount of data to reduce the network can also speed up the model of training, shorten the learning time.

Model Building
This paper adopts the most basic CNN model for deep learning. The established network model is trained and tested with data samples. Convolutional neural network is composed of a series of operation layers. In this paper, a total of 6 operation layers are used in the network, namely Conv2D, Activation, MaxPoolin92D, Ddropout, Dense, and Flatten.
Conv 2D is the convolution layer: the convolution operation is performed on the input tensor (usually the matrix of HxWxC). A convolution layer usually contains several convolution kernels. Each convolution check is used to carry out local weighting operation on the input image, and the parameters of weighting operation are learnable. The main function of the convolutional layer is to extract image feature maps. After the number of kernels passing through this convolutional layer, several feature maps will be generated. For example, feature map A may extract image edge features, feature map B may extract image color features, and so on.
A nonlinear operation is performed on the input tensor (usually the matrix of HXWXC). The nonlinear operation used in this network is mainly Max (x, 0), where x is each element in the matrix. The main function of the activation layer is that the samples are usually not linearly separable in the feature space, so it is necessary to carry out nonlinear transformation to make the sample points linearly separable in the transformed space.
MaxPooling2D is a pooling layer: the input tensor (usually the matrix of HXWXC) is locally maximized, equivalent to a local point to represent the entire region.
Dropout is the discard layer: Due to the large number of parameters and strong fitting ability of convolutional neural network, overfitting is easy to occur in the case of limited samples. Dropout is a means to reduce overfitting. The Dropout layer will randomly discard some elements (set to 0) according to a certain proportion of the input tensor. Since the discarded position is random each time, it requires the subsequent computing layer to use more global abstract features to classify, similar to a piece of incomplete jigsaw puzzle to obtain the overall information.
Flatten is the unwrapping layer: the input tensor (usually an HXWXC matrix) is unrolled into a one-dimensional vector (the feature that is ultimately used for classification).
Dense is the full connection layer: the input one-dimensional vector is linearly weighted to get the final classification probability.

Identification Process and Parameter Setting of Convolutional Neural Network
In this identification experiment, NewVGG neural network structure (JSON file) and corresponding weights (H5 file) were selected to reconstruct the neural network.
The recognition data set is divided into two parts. One part is 1000 lung cancer images and 1000 normal lung images selected from the public data set Deeplesion, a total of 2000 images. The other part is the desensitization CT data of the hospital. According to the PET-AC images corresponding to the CT, 200 CT images of suspected lung cancer are found for recognition. The format of the input image is 512*512 gray image of single channel.
The pre-processing Deeplesion recognition set uses the same processing method as the training data set, and finally makes a single channel image of 8bit grayscale in PNG format. In the preprocessing of hospital images, the Radiant Dicom Viewer (Radiant Dicom Viewer) is used to save the image of suspected lung cancer as a PNG format, and then the image is normalized.
Batch identification can be easily achieved by combining the recognition function with a simple cycle function. PREDICT outputs 0 as lung cancer and 1 as normal, and records the statistical accuracy.  Figure 1. Common dataset identification results As shown in Table 1 and Figure 1, since 2000 images are known to include 1000 lung cancers and 1000 normal lungs, ACC (TP)=89.4% and ACC (TN)=83.2% can be calculated through the accuracy formula of statistical recognition, and the recognition accuracy is 86.3%, namely, 274 images are judged wrong.

Results of Public Data Sets
Through observation, most of the misidentified images are not obvious after preprocessing. If lung window or mediastinal window can be used to make lung cancer more obvious, the recognition rate can be further improved. Therefore, in the recognition of hospital data, each image has been processed by adding window width and window level. The other part of data features have certain specificity, and the corresponding training sample size is relatively small, which leads to failure to learn the corresponding eigenvalues when training the network model. Most of the clinical manifestations of lung cancer are: vesicle sign, lobulation sign, burr sign, pleural depression sign, vascular cluster sign, small spinous process sign and calcification. Most of the image shapes such as vesicle sign, lobulation sign and burr sign can be well recognized.

Hospital Dataset Results
A total of 500 images were collected from the hospital dataset. The 500 patients were double-blind tested, and the images were confirmed with the assistance of doctors at the hospital.  Figure 2. Hospital dataset identification results As shown in Figure 2, in the 300 hospital images, the true positive rate was 85.3%, and the true negative rate was 53.5%. The combination of MSCT and magnetic resonance imaging (MRI) can reduce the missed diagnosis rate of lung cancer to 5 to 10 percent, compared with 15 to 20 percent for experienced physicians in clinical diagnosis. Through the analysis of the above test results, the missed diagnosis rate of the system in the public data set was 10.6%, and the missed diagnosis rate in the hospital data set was 14.7%, both lower than the clinical missed diagnosis rate. In this paper, the neural network is trained with public data set as training group and has been preprocessed to a certain extent. Compared with some hospital data, the specificity of some hospital data is strong. Among the 300 cases collected, there are some cases that do not have public data set. From the perspective of learning and research, this system can achieve a higher accuracy and a lower rate of missed diagnosis. From the perspective of clinical application, a large amount of hospital data is still needed to train the neural network to optimize the accuracy and missed diagnosis rate of the system.

Conclusions
With the continuous improvement of modern medical imaging technology and medical means, computer-aided diagnosis and computer-guided surgical treatment have been more and more applied in clinical practice. At the same time, it also put forward higher requirements for the corresponding computer technology, faster detection, more accurate location of the lesion area is the pursuit of the goal of the skill field. In this paper, CNN (Convolutional Neural Network) is used to design intelligent recognition of lung cancer. Its purpose is to provide an auxiliary diagnosis reference for examining doctors in the Department of Ultrasound and for imaging students in clinical practice. In the process of model design and training, image processing is adopted and the method of model building is used to improve the results of network classification. The final results show that deep learning can effectively identify lung cancer. Therefore, the application of deep learning in the clinical application of lung cancer has great prospects and is worth further exploration and research.


extfieldsofstudy:
Computer Science
Engineering
Medicine

text:
Addressing Limited Weight Resolution in a Fully Optical Neuromorphic Reservoir Computing Readout

Using optical hardware for neuromorphic computing has become more and more popular recently due to its efficient high-speed data processing capabilities and low power consumption. However, there are still some remaining obstacles to realizing the vision of a completely optical neuromorphic computer. One of them is that, depending on the technology used, optical weighting elements may not share the same resolution as in the electrical domain. Moreover, noise and drift are important considerations as well. In this article, we investigate a new method for improving the performance of optical weighting, even in the presence of noise and in the case of very low resolution. Even with only 8 to 32 levels of resolution, the method can outperform the naive traditional low-resolution weighting by several orders of magnitude in terms of bit error rate and can deliver performance very close to full-resolution weighting elements, also in noisy environments.

Photonic reservoir computing. Reservoir computing (RC) is one of the machine learning paradigms 10,11 whose relaxed requirements make it well suited for a hardware implementation. A reservoir is inherently a randomly initialized, untrained recurrent neural network (RNN), which acts as a temporal prefilter to transform a time dependent input into a higher-dimensional space, where it can be more easily classified by a linear classifier. Therefore, unlike RNNs, reservoir computing does not rely on optimizing the internal interconnection parameters of the network. Instead, it only optimizes the weights of the linear combination in the readout layer as shown in Fig. 1a. Because the performance of the reservoir is, within certain bounds, relatively insensitive to the exact internal weights of the reservoir, this technology is very interesting for a (photonic) hardware implementation, where fabrication tolerances are inevitable. Specific to photonic implementations are the so-called passive reservoirs, where the reservoir itself does not contain any nonlinearity, but where the photodetector in the readout (which converts a complex-valued light amplitude to a real-valued power intensity) provides the required non-linearity. Doing away with the need for internal nonlinearities reduces the power consumption. Photonic integrated circuits like these have been reported in 4,[12][13][14] , solving parity bit tasks, header recognition and telecom signal regeneration tasks at data rates around 10 Gb/s. Laporte et al. 5,15 use photonic crystal mixing cavities as the reservoir and also could solve the XOR task on neighboring bits, with the potential of achieving much higher speeds, theoretically up to several hundreds of Gb/s.
In the meantime, other neuromorphic computing approaches are gaining prominence. E.g. optical feedforward neural networks for deep learning are proposed by Shen et al. 7 , which cascade integrated coherent optical Readout system. To fully exploit a coherent optical neuromorphic computing circuit, thus benefiting from its high data rates and low energy consumption, an integrated optical readout to calculate the weighted sum ( Fig. 1b) is preferable. The critical aspect for such an optical readout is that the weights are applied in the optical domain, instead of converting optical signals to electrical signals, and then applying the weights on the electrical signals using a microcomputer (referred as electrical readout schemes as shown in Fig. 1c). Apart from speed, latency, and power efficiency advantages, such a fully optical readout system can also perform weighting on phase and amplitude separately, which results in more degrees of freedom, and thus more computation capability.
One issue of an optical readout is that there is no longer an explicit electronic observability for each of the states, since in such systems it is desirable to only have one single photodiode and high-speed analog-to-digital converter after the linear combination. This observability problem was dealt with by Freiberger et al. 17 .
Another problem is that, depending on the technology, the resolution of the optical weights could be much lower compared to applying weights in the electrical domain, which can easily reach 16 bit and more. For a photonic reservoir with optical readout illustrated as Fig. 1b, the situation can be different depending on the weights are physically implemented. One option for the OM (optical modulator) is a heater-based optical interferometer. By driving the heater current to produce temperature changes on an integrated waveguide, the phase and amplitude of the optical signal can be modified. In this case, the resolution of the current source will be the limitation of the weighting resolution and the weights will be quantized by the resolution of the DAC that drives the current source. So, in this case, weight resolution will not be a problem, but this heater-based implementation consumes a lot of power and is volatile.
Another more extreme example is a Barium Titanate (BTO) 18 weighting element, which has the critical advantage of being non-volatile, consuming nearly zero power while weighing the signal. Unlike the heater-based OM, we control the BTO material properties by programming it with a voltage, instead of changing the temperature. However, due to the stability of the ferroelectric properties of the material, there are only very few levels available for adjustment, therefore causing a very low resolution on the order of 3-5 bit (10-30 levels). Additionally, this is coupled with some inevitable noise due to potential parameter drifting of the optical elements.
In this paper, we will consider the extreme example of a BTO weighting element and explore strategy to solve the low weighting resolution problem in a general way.
Weight quantization. Network quantization is a important topic for deep learning researchers and developers, since the demand for applying machine learning tasks on low power consumption systems, e.g. mobile devices, has been increasing drastically 19 . However, almost all of the methods and algorithms have a focus on existing electrical hardware, like GPU or CPU. As hardware photonic neuromorphic computing networks have been thriving recently, and since the weighting method is completely different, it needs its own quantization strategy that suits its features and maximizes the performance from optical weighting elements. The differences are reflected in the following aspects.
Weighting resolution. First, for deep learning network quantization in the electrical domain, the target resolution is mostly based on the application hardware. Vanhoucke et al. 20 implemented an 8-bit integer resolution activation instead of 32-bit floating-point to optimize CPU performance. Courbariaux et al. 21   www.nature.com/scientificreports/ weights to accelerate future dedicated low-power electrical deep learning hardware by replacing multiply-accumulate operations by simple accumulations, which is called BinaryConnect method. However, for photonics, getting rid of multiplications is not so relevant, as these can be done in the analog optical domain instantly anyhow. The BinaryConnect method was later extended to a power-of-two setting by the Gudovskiy et al. 22 , but even then, the peculiarities of photonic hardware are such that a new discretisation strategy is required. Indeed, in integrated photonics systems weighting is usually achieved by controlling on-chip heaters or non-volatile materials in an optical interferometer to modulate the amplitude or phase of the coherent optical signal. Depending on the technology, the resolution varies between 3 and 6 bit, but more importantly, it is often impossible to achieve a weight identical to zero. This aspect needs to be accounted for explicitly, and hence a new discretization strategy is required.

Number of parameters.
Most deep neural networks have a substantial number of trainable parameters; some of them may reach tens of millions, leading to a significant redundancy in deep learning models 23 . Network pruning is often a feasible option for reducing the size of the network in parallel to quantization. However, for optical systems like passive reservoir computing chips, the number of trainable parameters in the readout system is typically in the hundreds or fewer, which leads to a reduced parameter redundancy and therefore a potentially much more severe impact on the performance by the weight quantization. On the other hand, each trainable parameter is corresponding to a real optical channel, which also makes pruning not possible, since it would result in additional optical loss.
Quantization details. As mentioned above, weighting in a coherent photonic neuromorphic hardware is different compared to weighting in a CPU or GPU. One of the differences lies in the nature of the quantized weight values. In 21 , the quantized weights can be chosen from 1 or − 1, and for the majority of deep learning networks, there is no restriction on the distribution of the weight values. However, for amplitude weighting in the coherent photonic neuromorphic hardware, it is not possible to achieve a weight value of exactly 0, due to inevitable hardware imperfection. As a example, an optical interferometer has a ' on' state and an ' off ' state, but because of fabrication imperfections causing loss imbalance, the ratio between the intensity of the ' on' state and the intensity of the ' off ' states can never reach infinity. This ratio is so-called extinction ratio of highest to lowest weight, which is an important implementation aspect. For phase weighting on the other hand, there is no such restriction, and the values can range from 0 to 2π.
Noise. Another difference is that for quantization in a CPU-or GPU-based deep networks, since they are digital systems, there should be negligible noise on the quantization weights. Although some networks apply noise on the weights to improve the robustness of the system 24 , the added noise is often really small compared to the quantization interval. In contrast, in hardware optical weighting elements, noise is inevitable, and can sometimes be significant, which represents additional challenges. Due to the many intrinsic differences between a photonic weighting system and an electrical weighting mechanism as discussed above, it is very difficult to apply a deep learning quantization method directly to photonic hardware. In such a context, it is of great importance to train a readout system for coherent optical neuromorphic computing systems, so that the performance is less sensitive to the quantization and the noise of the weights. This is the exact problem we want to tackle in this paper.
The rest of this paper is structured as follows. In "Methods" section, we introduce our modeling method of the optical weighting hardware, in amplitude and phase, by taking into consideration the number of quantization levels, the extinction ratio of the elements, and the noise after quantization. The explorative quantization retraining technique used in this work will also be highlighted. In "Results" section, we show our simulation results based on our quantization method with respect to the different hardware parameters mentioned in "Methods" section. We also discuss the quantization result for various tasks.

Methods
In this part, we introduce the concept of our quantization method and present the implementation details like the discretization model, the type of regularization and the training procedure.
Explorative quantization weight selection. The explorative quantization weight selection in this work is inspired by methods that have been used in deep learning quantization. Typically, after a full precision model has been trained, a subset of weights is identified to be either pruned 25 or kept fixed 26 . The other weights are then retrained in full precision and requantized. If necessary, this step can be repeated in an iterative fashion, retraining progressively smaller subsets of the weights in order to find the most optimal and stable solution.
A crucial part of these methods is selecting a subset of weights to be left fixed or to be pruned. Random selection of weights is not a good idea, because there is a high probability of eliminating 'good' weights that convey important information. Han et al. 25 and Zhou et al. 26 tackle this problem by choosing the weights with the smallest absolute value. This is reasonable in deep learning models, since the millions of weights can provide enough tolerance when it comes to accidentally selecting the 'wrong' weights. However, in the readout systems we are investigating here, we have much fewer weights and a much more limited resolution with severe noise. In this case, the absolute value will not provide enough information, as a combination of many small weights could be important in finetuning the performance of the network. This will lead to a risk of a huge accuracy loss when specific 'wrong' connections (that are more sensitive to perturbations) are chosen to be retrained. www.nature.com/scientificreports/ Instead, we adopt a different (albeit more time-consuming) approach. The specific procedure is as follows. We start with a pre-trained model, which consists of all the full-resolution weights and perform a nearest rounding quantization on the complex weights. After quantization, the weights are divided into two groups, one of them will be retrained and the other group not. This partition of the weights in groups is done not once, but several times in parallel by random selection (the number of parallel weights partitions is chosen from 2 to 20 depending on the number of residual weights to be retrained). For each partition, after retraining, the weights will be quantized again to evaluate the performance based on the BER on the training data set. By comparing the task performance for these different partitions, we can pick the best one.
We iteratively conduct the procedure above. In each iteration, the ratio of weights to be retrained decreases by a factor of two, starting from an initial value of 0.5. We typically perform 4 iterations, so for our 16-node reservoir, the number of trained weights in each iteration is 8, 4, 2, 1. As the iteration proceeds, we also decrease the number of partitions to be evaluated from 20 in the beginning to 2 at the end, as fewer and fewer weight combinations become available. In order to increase the stability of the procedure, weights that have been selected as fixed in previous iterations remain fixed in future iterations as well.
Quantization of the optical weighting elements. Here, we describe in more detail what model has been used for the quantization procedure itself. We aim to provide a high-level model for the weighting elements (both amplitude and phase) that is as generic as possible, without being tied to any specific hardware implementation details. Rather, in the quantization process, we take three major aspects into account: extinction ratio, resolution, and noise. Figure 2a shows the quantization of complex weights in the complex plane, without weighting noise.
Amplitude quantization. The readout systems that we model here are supposed to be an optical system without amplification. In this case, the largest weight of the amplitude will be 1 (light passing through without being affected) and the lowest weight will be 0 (light is fully blocked by the weighting elements). However, in a realistic optical weighting element, the fully blocked weight '0' is often not achievable. Most elements have an extinction ratio describing how much optical power it can block compared to the maximum power that can pass through. For example, given an an extinction ratio of 10 and a maximum weight of 1, the minimum weight will be 0.1.
The resolution of the weights is characterized as a number of bits or number of levels. All the possible weighting levels are uniformly distributed within the same interval, taking into account the minimum weight because of the extinction ratio. Figure 2b illustrates this in more detail. The minimum weight the readout system can reach is given by: After quantization, weights that are below this level will be rounded up to this value. The distance between two adjacent weighting values is given by: where N is the number of available weighting levels, determined by the resolution of the optical element.  www.nature.com/scientificreports/ Noise and drift are of course very dependent on the details of the hardware used, but for our purposes, we will abstract the noise as a Gaussian distribution. The noise level numbers we give in the following section are based on the standard deviation σ of the Gaussian distribution as follows: Phase quantization. In the phase quantization process, the extinction ratio issue will not play a role, and there will only be resolution and noise. The quantization will be implemented between [0, 2π) . All the levels are also evenly distributed. Noise is also modeled as Gaussian distribution.
Regularization. The regularization method used in this work is L2 regularization. In the photonic hardware readout system, there is no significant redundancy of connections or weights, and each weight is realized by a real waveguide through which the optical signals are passing. It is not practical to ' cut' or 'prune' any hardware optical connections. Therefore we choose L2 over L1 regularization since we do not want to waste any readout waveguide and L1 tends to result in zero weights. Choosing the best regularization parameter is crucial to prevent overfitting and to help the system to be robust against noise and quantization 27 .
Baselines. In this paper, we choose two baseline methods to compare our results with. The first is nearest rounding quantization, which is referred to as 'naive quantization' in the rest of the paper. The second baseline is randomized rounding quantization 28 , which will be referred to as 'random rounding' . Random rounding is a reasonable baseline to compare our method too, since it also gives an optimized performance by statistic selection of better weight choices. In our implementation of random rounding, we set the rounding probability based on the distance of a weight to a nearby quantization value. We repeat this procedure 20 times and pick the result with the lowest error rate.
Training procedure. The original idea of reservoir computing is to use linear (ridge) regression to train the weights of a readout system. This saves a lot of training time compared to using gradient descent, as weighing the reservoir output signals is just a linear regression and can be calculated explicitly. In a coherent photonics reservoir however, the outputs of the reservoir are coherent optical signals containing amplitude and phase information, i.e. complex numbers. In the readout system, these optical signals are mixed together and interfere with each other to produce a final optical signal injected into a photodetector. Therefore the output of such a readout system is a single electrical signal containing only intensity information. This is a nonlinear conversion from a complex-valued optical signal with an amplitude and phase to a purely real intensity. To incorporate such a nonlinear conversion, we can no longer use the one-step solution based on the Moore-Penrose inverse of linear ridge regression, since there is no unique way to choose a phase of the signal before the photodiode that gives rise to the desired amplitude after the photodiode. Therefore, in our learning phase, we optimize the weights with gradient descent, explicitly taking the nonlinear photodiode into account. Using this method gives us more robust results and increases the computation capability of our reservoir system.
Another decision we made is to use the Mean Squared Error (MSE) as a loss function to optimize the optical weights. One could argue to use logistic regression instead, because the tasks we are trying to solve are all classification tasks. However, to build an optical neuromorphic computing system, we have to consider the hardware implementation of each signal processing step. In our hardware system, the output electrical signal from the final photodetector will be used as the final prediction signal without further signal processing. Using logistic regression would assume another layer of nonlinear calculation, the sigmoid function. This process would result in a more complex integration technology and induce more latency and energy cost.

Results
In this section, we present simulation results for the performance of the proposed method on two tasks: header recognition and a boolean XOR operation. The bit sequences will be encoded to optical signals before the simulation. The architecture we choose for these tasks is the 4-port swirl reservoir network 29 with 16 nodes. We train all the tasks on fully integrated passive photonic reservoirs with optical integrated readout systems. We treat the full resolution weighting performance with weighting noise as the idealized best performance the system can achieve, and also compare our method with the performance of 'naive' quantization and random rounding quantization.
The simulation consists of two parts. The first one is to generate the reservoir signal. We use Caphe 30 to simulate the reservoir circuit with subsampling of 20 points per bit on the intensity-modulated input signal. The delays in the photonic reservoir are optimized for an input signal speed of 32 Gbps. The response of the photonic reservoir will be the complex output signals from each of the nodes, consisting of amplitude and phase information. The second part is the training and quantization of the weights using our proposed method. We use PyTorch 31 to implement this.
Header recognition. The header we use in the header recognition task is a 4-bit header '1101' . We want the final readout signal to be '1' whenever the chosen header appears in the input signal, and '0' otherwise. The reservoir architecture is a 4 × 4 , 16 node reservoir. Although header recognition can be solved in a multi-class classifier, the optical readout hardware we use here is not suited for multi-channel optical output. As mentioned, we use an MSE loss function with L2 regularization and train our models with gradient descent using Adam 32 .
We now discuss the influence of resolution, noise and extinction ratio on the performance. For the following figures indicating the performance in terms of BER, error bars represent the standard deviation of the results Noise. Noise has a very significant influence on performance. Figure 3 shows the performance evaluation of the Bit Error Rate (BER) at different noise levels at 8-level, 16-level and 32-level resolution on the 4-bit header recognition task. The top figure with 8-level resolution shows that overall, the quantized weights obtained by our explorative retraining method consistently provide performances very close to the full resolution weights. For more substantial noise levels of the optical weighting elements, as shown in the far right part of the graphs, the BER increases significantly, which reflects the severe impact that the noise of the optical weighting elements brings to system performance, even for a full-resolution system. Apart from those situations of large noise levels, the retrain method is performing well throughout and gives several orders of magnitude better BER than the 'naive' quantization of the weights where the readout is trained and quantized only once. When there is no noise on the weights, the retrain method gives almost the same performance on the limited resolution readout system compared to a full resolution readout system. This result is rather surprising, given that the resolution here is only 8 levels. The random rounding method also fails to deliver a reasonable performance at these levels of resolution. For higher resolutions, namely 16 levels and 32 levels, the explorative retraining method provides even closer performance to the full resolution readout systems. As the resolution increases, the 'naive' quantization method also achieves lower bit error rates, however still not comparable to the full resolution and explorative retrained weights.Random rounding only provides comparable performance to the full resolution weights at 32-level resolution. For lower resolutions, its performance is considerably worse. When the noise levels are high, all four sets of weighting strategies are giving a similar performance (especially at 16 and 32 levels resolution all four results are indistinguishable), The reason is that, as the resolution increases, the interval between two adjacent weighting level is in the same range of the variance of the noise. These three weighting methods will provide similar weighting values statistically.
Extinction ratio. Figure 4 illustrates the influence of the extinction ratio of the optical weighting elements. The figure shows that for directly quantized weights, a higher resolution always gives better BER regardless of the extinction ratio of the weighting components. However, when the extinction ratio gets higher, the perfor-  www.nature.com/scientificreports/ mance from the 32-level 'naive' quantized weights and random rounding weights are closer to the full resolution weights. This result is intuitive since higher extinction ratio means the weight range is more preserved after quantization, and when the extinction ratio is low, more initially lower weights will be rounded up to the lowest weighting value that the extinction ratio allows. In this task, extinction ratios of 5.0 and 10.0 do not give much difference compared to the huge performance drop for the extinction ratio of 2.0. For photonics weighting components, an extinction ratio of 5.0 to 10.0 is not a very strict requirement, and therefore photonics reservoir hardware systems should be able to deliver the required extinction ratio for this task.
In contrast, the retraining method gives consistently better performance because the weights are continually evolving to adapt to the weighting range that the extinction ratio defines. The performance levels are always very close to those of the full resolution, with a limited impact from the extinction ratio.
Training time. The training time of the explorative method takes longer due to iterations of training and weight connection selecting. For the header recognition task, training the 16-node photonic reservoir system using nearest rounding takes around 2 minutes, with CUDA enabled on a Nvidia GeForce GTX 970 GPU. The explorative training procedure takes around 18 minutes and random rounding method takes 5 minutes. For a reservoir that has more nodes, the training time does not scale up quickly with the number of nodes, since the training utilizes the parallel processing of the GPU. Moreover, as mentioned, the number of nodes in a photonics reservoir is usually below one hundred to control optical loss.
Although it is a more time-consuming training procedure, the extra training time does not impact to the run time the system after programming. As mentioned, the training of the system is done off-line. Once the system is trained, the photonic reservoir computing works in the all-optical domain without any optical-electrical conversion and can have an extremely high bandwidth. Therefore the off-line training time does not result in any runtime latency.

4-Bit delayed XOR.
We chose 4-bit delayed XOR (i.e. calculating the XOR of the current bit and 4 bits ago) as our second task. The reason is that XOR is a more nonlinear task compared to the header recognition, so it is interesting to see how it behaves under weight quantization situations. Figure 5 presents the performance of our quantization retraining method on low-precision weights (8, 16, 32 levels) at different noise levels. From the blue curve, it can be seen that 4-bit delayed XOR is indeed a harder task to tackle since even with full precision, the BER increases significantly with a small amount of noise. Moreover, the task is also very sensitive to low precision weights, as shown from the 'naive' quantization weights (orange curves), at the 8 levels resolution, the tasks is unsolvable even without any noise involved. In the meantime, we also observe that our explorative retrained weights are capable of providing very close performance to full resolution weights, except for 8 levels resolution, where due to the extra nonlinearity requirement of the task, retrained weights find themselves hard to follow. But still, 'naive' direct quantized weights and random rounding quantized weights are severely outperformed by our explorative retraining method at the overall spectrum of the different noise levels.
The performance dependence on resolution and extinction ratio is shown in Fig. 6. The noise level here is 0.0025. Similar to the header recognition task, a higher extinction ratio provides better direct quantization performance. Weighting elements with an extinction ratio of 2 are not able to provide enough weighting range for direct quantized weights to deliver workable performance regardless of the weighting resolution. The explorative retraining method is also affected at the 8 levels resolution. When the extinction ratio is 10, the quantization retraining method delivers performance close to the full resolution, from which we can draw a statement that an extinction ratio of 10 is sufficient for this task. For random rounding weights, it always delivers slightly better results compared to the 'naive' quantization method. However, it is outperformed by the explorative retraining method by a big margin, especially in higher resolutions regardless of extinction ratios. www.nature.com/scientificreports/

Conclusion
In this paper, we addressed the limited weight resolution in an integrated all-optical readout system for photonic reservoir computing systems. A high-level general modeling for realistic integrated optical weighting elements enables us to characterize the influence of the weighting resolution, noise, and extinction ratio on the system performance. Our proposed explorative retraining method focused on identifying the best weights to be retrained. It is shown that in situations where both the number of output channels and the weighing resolution is extremely limited, our proposed method still delivers performance very close to that of full resolution weights over a large span of weighting noises.
Received: 24 August 2020; Accepted: 20 January 2021 Figure 6. Performance comparison of three different extinction ratios, 2 (left), 5 (middle), 10 (right) for the 4-bit delayed XOR task over different resolutions. The blue curve represents the performance of full-resolution weights; the orange curve represents 'naive' quantization weights; the green curve for explorative retrained quantization weights.


Measuring the Contribution of Multiple Model Representations in Detecting Adversarial Instances

Deep learning models have been used for a wide variety of tasks. They are prevalent in computer vision, natural language processing, speech recognition, and other areas. While these models have worked well under many scenarios, it has been shown that they are vulnerable to adversarial attacks. This has led to a proliferation of research into ways that such attacks could be identified and/or defended against. Our goal is to explore the contribution that can be attributed to using multiple underlying models for the purpose of adversarial instance detection. Our paper describes two approaches that incorporate representations from multiple models for detecting adversarial examples. We devise controlled experiments for measuring the detection impact of incrementally utilizing additional models. For many of the scenarios we consider, the results show that performance increases with the number of underlying models used for extracting representations.

Introduction
Research on neural networks has progressed for many decades, from early work modeling neural activity (McCulloch and Pitts 1943) to the more recent rise of deep learning (Bengio, Lecun, and Hinton 2021). Notable applications include image classification (Krizhevsky, Sutskever, and Hinton 2012), image generation , image translation (Isola et al. 2017), and many others (Dargan et al. 2020). Along with the demonstrated success it has also been shown that carefully crafted adversarial instances-which appear as normal images to humanscan be used to deceive deep learning models (Szegedy et al. 2014), resulting in incorrect output. The discovery of adversarial instances has led to a broad range of related research including 1) the development of new attacks, 2) the characterization of attack properties, and 3) defense techniques. Akhtar and Mian present a comprehensive survey on the threat of adversarial attacks to deep learning systems used for computer vision.
Two general approaches-discussed further in Section 6-that have been proposed for defending against adversarial attacks include 1) the usage of model ensembling and 2) the incorporation of hidden layer representations as discriminative features for identifying perturbed data. Building on these ideas, we explore the performance implications that can be attributed to using representations from multiple models for the purpose of adversarial instance detection.
Our Contribution In Section 3 we present two approaches that use neural network representations as features for an adversarial detector. For each technique we devise a treatment and control variant in order to measure the impact of using multiple networks for extracting representations. Our controlled experiments in Section 4 measure the effect of using multiple models. For many of the scenarios we consider, detection performance increased as a function of the underlying model count.

Preliminaries
Our research incorporates l-layer feedforward neural networks, functions h : X → Y that map input x ∈ X to output y ∈ Y through linear preactivation functions f i and nonlinear activation functions φ i .
The models we consider are classifiers, where the outputs are discrete labels. For input x and its true class label y, let J(x, y) denote the corresponding loss of a trained neural network. Our notation omits the dependence on model parameters θ, for convenience.

Adversarial Attacks
Consider input x that is correctly classified by neural network h. For an untargeted adversarial attack, the adversary tries to devise a small additive perturbation ∆x such that adversarial input x adv = x + ∆x changes the classifier's output (i.e., h(x) = h(x adv )). For a targeted attack, a desired value for h(x adv ) is an added objective. In both cases, the L p norm of ∆x is typically constrained to be less than some threshold . Different threat models-white-box, grey-box, and black-box-correspond to varying levels of knowledge that the adversary has about the model being used, its parameters, and its possible defense.
The adversary's objective can be expressed as an optimization problem. For example, the following constrained maximization of the loss function is one way of formulating how an adversary could generate an untargeted adversarial arXiv:2111.07035v2 [cs.
LG] 12 Feb 2022 There are various ways to generate attacks. Under many formulations it's challenging to devise an exact computation of ∆x that optimizes the objective function. An approximation is often employed.
Fast Gradient Sign Method (FGSM) (Goodfellow, Shlens, and Szegedy 2015) generates an adversarial perturbation ∆x = · sign(∇ x J(x, y)), which is the approximate direction of the loss function gradient. The sign function bounds its input to an L ∞ norm of 1, which is scaled by .
Basic Iterative Method (BIM) (Kurakin, Goodfellow, and Bengio 2017) iteratively applies FGSM, whereby The L ∞ norm is bounded by α on each iteration and by t · α after t iterations. x adv t can be clipped after each iteration in a way that constrains the final x adv to an -ball of x.
Carlini & Wagner (CW) (Carlini and Wagner 2017) generates an adversarial perturbation via gradient descent to solve ∆x = argmin δ ( δ p + c · f (x + δ)) subject to a box constraint on x + δ. f is a function for which f (x + δ) ≤ 0 if and only if the target classifier is successfully attacked. Experimentation yielded the most effective f -for targeted attacks-of those considered. c is a positive constant that can be found with binary search, a strategy that worked well empirically. Clipping or a change of variables can be used to accommodate the box constraint.

Ensembling
Our research draws inspiration from ensembling, the combination of multiple models to improve performance relative to the component models themselves. There are various ways of combining models. An approach that is widely used in deep learning averages outputs from an assortment of neural networks; each network having the same architecture, trained from a differing set of randomly initialized weights.

Method
To detect adversarial instances, we use hidden layer representations-from representation models-as inputs to adversarial detection models. For our experiments in Section 4, the representation models are convolutional neural networks that are independently trained for the same classification task, initialized with different weights. Representations are extracted from the penultimate layers of the trained networks. The method we describe in this section is more general, as various approaches could be used for preparing representation models. For example, each representation model could be an independently trained autoencoderas opposed to a classifier-with representations for each model extracted from arbitrary hidden layers. Additionally, it's not necessary that each of the models-used for extracting representations-has the same architecture.
We devise two broad techniques-model-wise and unitwise-for extracting representations and detecting adversarial instances. These approaches each have two formulations, a treatment that incorporates multiple representation models and a control that uses a single representation model. For each technique, the functional form of the detection step is the same across treatment and control. This serves our objective of measuring the contribution of incrementally incorporating multiple representation models, as the control makes it possible to check whether gains are coming from some aspect other than the incorporation of multiple representation models.
The illustrations in this section are best viewed in color.

Model-Wise Detection
With N representation models, model-wise detection uses a set of representations from each underlying model as separate input to N corresponding detection models that each outputs an adversarial score. These scores, which we interpret as estimated probabilities, are then averaged to give an ensemble adversarial probability estimate. A baselineholding fixed the number of detectors-uses a single representation model as a repeated input to multiple detection models. The steps of both approaches are outlined below.

Model-Wise Treatment
Step 1 Extract representations for input x from N representation models. ...

x x x
Step 2 Pass the Step 1 representations through N corresponding detection models that each output adversarial probability (denoted P i for model i).
Step 3 Calculate adversarial probability P as the average of Step 2 adversarial probabilities.

Model-Wise Control
Step 1 Extract representations for input x from a single representation model. x Step 2 Pass the Step 1 representations through N detection models that each outputs adversarial probability (denoted P i for model i). P 1 P 2 P N ...
Step 3 Calculate adversarial probability P as the average of Step 2 adversarial probabilities.

Unit-Wise Detection
With N representation models, unit-wise detection incorporates a single representation from each underlying model to form an N -dimensional array of features as input to a single detection model. A baseline-holding fixed the number of features for the detector-uses a set of units from a single representation model to form an input array for a detection model. The steps of both approaches are outlined below.

Unit-Wise Treatment
Step 1 Extract a single representation for input x from N representation models. There is no requirement on which unit is selected nor whether there is any correspondence between which unit is selected from each model.
Step 2 Pass the N -dimensional array of Step 1 representations through an adversarial detection model that outputs adversarial probability P . P

Unit-Wise Control
Step 1 Extract N units from the representations for input x from a single representation model. In the illustration that follows, the count of extracted representation units, N , matches the total number of units available. It's also possible for N to be smaller than the quantity available. x Step 2 Pass Step 1 representations through an adversarial detection model that outputs adversarial probability P . P

Measuring the Contribution from Multiple Models
We are interested in measuring the contribution of multiple models for detecting adversarial instances. For both the model-wise and unit-wise detection techniques, the contribution of multiple models can be evaluated by inspecting the change in treatment performance when incrementing the number of representation models, N . The changes should be considered relative to the control performance, to check whether any differences are coming from some aspect other than the incorporation of multiple representation models.

Experimental Settings
We conducted experiments using the CIFAR-10 dataset (Krizhevsky 2009), which is comprised of 60,000 32×32 RGB images across 10 classes. The dataset, as received, was already split into 50,000 training images and 10,000 test images. We trained one neural network classifier that served as the target for generating adversarial attacks. We trained 1,024 additional neural network classifiers to be used as representation models-with representations extracted from the 512-dimensional penultimate layer of each network. A different randomization seed was used for initializing the weights of the 1,025 networks. Each network had the same-18-layer, 11,173,962-parameter-ResNetinspired architecture, with filter counts and depth matching the kuangliu ResNet-18 architecture. 1 Pixel values of input images were scaled by 1/255 to be between 0 and 1. The networks were trained for 100 epochs using an Adam optimizer (Kingma and Ba 2014), with random horizontal flipping and random crop sampling on images padded with 4 pixels per edge. The model for attack generation had 91.95% accuracy on the test dataset. The average test accuracy across the 1,024 additional networks was 92.22% with sample standard deviation of 0.34%.
Adversarial Attacks Untargeted adversarial perturbations were generated for the 9,195 images that were originally correctly classified by the attacked model. Attacks were conducted with FGSM, BIM, and CW, all using the cleverhans library (Papernot et al. 2018). After each attack, we clipped the perturbed images between 0 and 1 and quantized the pixel intensities to 256 discrete values. This way the perturbed instances could be represented in 24-bit RGB space. For FGSM, we set = 3/255 for a maximum perturbation of 3 intensity values (out of 255) for each pixel on the unnormalized data. Model accuracy on the attacked modelfor the 9,195 perturbed images-was 21.13% (i.e., an attack success rate of 78.87%). Average accuracy on the 1,024 representation models was 61.69% (i.e., an attack transfer success rate of 38.31%) with sample standard deviation of 1.31%.
For BIM, we used 10 iterations with α = 1/255 and maximum perturbation magnitude clipped to = 3/255. This re-  Figure 1: Example CIFAR-10 images after adversarial perturbation. The original image-in the leftmost column-is followed by three columns corresponding to FGSM, BIM, and CW attacks, respectively. Images were chosen randomly from the set of test images that were correctly classified without perturbation-the population of images for which attacks were generated.
sults in a maximum perturbation of 1 unnormalized intensity value per pixel on each step, with maximum perturbation after all steps clipped to 3. Accuracy after attack was 0.61% for the attacked model. Average accuracy on the 1,024 representation models was 41.09% with sample standard deviation of 2.64%. For CW, we used an L 2 norm distance metric along with most default parameters-a learning rate of 0.005, 5 binary search steps, and 1,000 maximum iterations. We raised the confidence parameter 2 to 100 from its default of 0, which increases attack transferability. This makes our experiments more closely align with black-box and grey-box attack scenarios, where transferability would be an objective of an adversary. Accuracy after attack was 0.07% for the attacked model. Average accuracy on the 1,024 representation models was 5.86% with sample standard deviation of 1.72%. Figure 1 shows examples of images that were perturbed for our experiments. These were chosen randomly from the 9,195 correctly classified test images-the population of images for which attacks were generated.

Adversarial Detectors
We use the 512-dimensional representation vectors extracted from the 1,024 representation models as inputs to model-wise and unit-wise adversarial detectors-both treatment and control configurationsas described in Section 3. All detection models are binary classification neural networks that have a 100-dimensional hidden layer with a rectified linear unit activation function. We did not tune hyperparameters, instead using the defaults as specified by the library we employed, scikit-learn (Pedregosa et al. 2011). Model-wise detectors differed in their randomly initialized weights.
To evaluate the contribution of multiple models, we run experiments that vary 1) the number of detection models used for model-wise detection, and 2) the number of units used for unit-wise detection. For the treatment experiments, the number of underlying representation models matches 1) the number of detection models for model-wise detection and 2) the number of units for unit-wise detection. For the control experiments, there is a single underlying representation model.
The number of units for the unit-wise control models was limited to 512, based on the dimensionality of the penultimate layer representations. The number of units for the unitwise treatment was extended beyond this since its limit is based on the number of representation models, for which we had more than 512. One way to incorporate more units into the unit-wise control experiments would be to draw units from other network layers, but we have not explored that for this paper.
We are interested in the generalization capabilities of detectors trained with data from a specific attack. While the training datasets we constructed were each limited to a single attack algorithm, we separately tested each model using data attacked with each of the three algorithms-FGSM, BIM, and CW.
For training and evaluating each detection model, the dataset consisted of 1) the 9,125 images that were originally correctly classified by the attacked model, and 2) the 9,125 corresponding perturbed variants. Models were trained with 90% of the data and tested on the remaining 10%. Each original image and its paired adversarial counterpart were grouped, i.e., they were never separated such that one would be used for training and the other for testing.
We retained all 9,125 perturbed images and handled them the same (i.e., they were given the same class) for training and evaluation, including the instances that did not successfully deceive the attacked model. For BIM and CW, the consequence of this approach is presumably minor, since there were few unsuccessful attacks. For FGSM, which had a lower attack success rate, further work would be needed to 1) study the implications and/or 2) implement an alternative approach.
We conducted 100 trials for each combination of settings. For each trial, random sampling was used for 1) splitting data into training and test groups, 2) choosing representation models, and 3) choosing which representation units to use for the unit-wise experiments.

Hardware and Software
The experiments were conducted on a desktop computer running Ubuntu 21.04 with Python 3.9. The hardware includes an AMD Ryzen 9 3950X CPU, 64GB of memory, and an NVIDIA TITAN RTX GPU with 24GB of memory. The GPU was used for training the CIFAR-10 classifiers and generating adversarial attacks. The code for the experiments is available at https://github. com/dstein64/multi-adv-detect.

Results
Model-Wise Figure 2 shows average model-wise adversarial input detection accuracies-calculated from 100 trials-plotted across the number of detection models. The subplots represent different combinations of training data attacks and test data attacks. The endpoint values underlying the figure are provided in the appendix.
Unit-Wise Figure 3 shows average unit-wise adversarial input detection accuracies-calculated from 100 trialsplotted across the number of units. The subplots represent different combinations of training data attacks and test data attacks. The endpoint values underlying the figure are provided in the appendix.

Discussion
Although subtle, for most scenarios the model-wise control experiments show an upward trend in accuracy as a function of the number of detection models. This is presumably an ensembling effect where there are benefits from combining multiple detection models even when they're each trained on the same features. The model-wise treatment experiments tend to outpace the corresponding controls, highlighting the benefit realized when the ensemble utilizes representations from distinct models.
The increasing accuracy for the unit-wise control experiments-as a function of the number of units-is more discernible than for the corresponding model-wise control experiments (the latter being a function of the number of models). The unit-wise gains are from having more units, and thus more information, as discriminative features for detecting adversarial instances. In most scenarios the treatment experiments-which draw units from distinct representation models-have higher performance than the corresponding controls. An apparent additional benefit is being able to incorporate more units when drawing from multiple models, not limited by the quantity of eligible units in a single model. However, drawing units from multiple models also comes at a practical cost, as it requires more computation relative to  drawing from a single model. As expected, detectors trained with data from a specific attack perform best when tested with data from the same attack. Interestingly, detectors trained with BIM attack data appear to generalize better relative to detectors trained with FGSM or CW attack data. This may be related to the hyperparameters we used for each of the attacks, as opposed to being something representative of BIM more generally.

Related Work
We are aware of two general research areas that are related to what we've explored in this paper. The approaches include 1) the incorporation of ensembling for adversarial defense, and 2) the usage of hidden layer representations for detecting adversarial instances.

Ensembling-Based Adversarial Defense
Combining machine learning models is the hallmark of ensembling. For our work, we trained detection models that process representations extracted from multiple independently trained models. For model-wise detection, we averaged detection outputs across multiple models. Existing research has explored ensembling techniques in the context of defending against adversarial attacks . Bagnall, Bunescu, and Stewart train an ensemble-to be used for the original task, classification, and also for adversarial detection-such that the underlying models agree on clean samples and disagree on perturbed examples. The adaptive diversity promoting regularizer (Pang et al. 2019) was developed to increase model diversity-and decrease attack transferability-among the members of an ensemble. Abbasi et al. devise a way to train ensemble specialists and merge their predictions-to mitigate the risk of adversarial examples.

Attack Detection from Representations
For our research we've extracted representations from independently trained classifiers to be used as features for adversarial example detectors. Hidden layer representations have been utilized in various other work on adversarial instance detection. Neural network invariant checking (Ma et al. 2019) detects adversarial samples based on whether internal activations conflict with invariants learned from nonadversarial data. Wójcik et al. use hidden layer activations to train autoencoders whose own hidden layer activationsalong with reconstruction error-are used as features for attack detection. Li and Li develop a cascade classifier that incrementally incorporates statistics calculated on convolutional layer activations. At each stage, the instance is either classified as non-adversarial or passed along to the next  stage of the cascade that integrates features computed from an additional convolutional layer. In addition to the methods summarized above, detection techniques have also been developed that 1) model the relative-positioned dynamics of representations passing through a neural network (Carrara et al. 2019), 2) use hidden layer activations as features for a k-nearest neighbor classifier (Carrara et al. 2017), and 3) process the hidden layer units that were determined to be relevant for the classes of interest (Granda, Tuytelaars, and Oramas 2020).

Conclusion and Future Work
We presented two approaches for adversarial instance detection-model-wise and unit-wise-that incorporate the representations from multiple models. Using those two approaches, we devised controlled experiments comprised of treatments and controls, for measuring the contribution of multiple model representations in detecting adversarial instances. For many of the scenarios we considered, experiments showed that detection performance increased with the number of underlying models used for extracting representations.
The research leaves open various avenues for future work.
• For our experiments, we trained 1,024 neural network representation models, whose diversity arises from using a different randomization seed for each. Perhaps other methods for imposing diversity would impact the performance of the detectors that depend on those models. • It would be interesting to explore how existing adversarial defenses fare when extended to use multiple underlying models. • Although we evaluated detectors across different attack algorithms, we always used data from a single attack for the purpose of training. Future research could investigate the effect of training with data from multiple attacks and/or varying hyperparameter settings for a specific attack. • Our focus was on measuring the incremental gains of detecting attacks when incorporating multiple representation models. Further work could perform a thorough defense evaluation under more challenging threat models.


extfieldsofstudy: Physics Psychology

text:
Time domain analysis of electroencephalogram (EEG) signals for word level comprehension in deaf graduates with congenital and acquired hearing loss

Deafness can be classified on the basis of onset as congenital and acquired hearing loss. The brain is a sensitive part of our body, electrical pulses from the neurons interact with each other, generating brain signals. EEG signals are extensively used for clinical diagnosis for any brain anomalies, language comprehension and performance measurement studies. This study mainly focuses on analysing the word level comprehension in deaf adults in the age group (21 -25 years) using EEG signals. The raw EEG signals were pre-processed and the relevant time domain linear and nonlinear features were extracted and classified using machine learning algorithms. The approximate entropy feature was found to be best suited for finding the comprehension of both congenital and acquired deaf adults. This feature of ISL was observed to be achieving better classification rate with a maximum average accuracy of 96% in both congenital and acquired deaf adults using SVM classifier.

1.Introduction
Hearing is a complex function of ear which allows discrimination of sounds. Hearing plays a key role in facilitating the language and speech advance [1]. As the table 1 shows, Goodmann classified deafness on the basis of the hearing impairment severity [2]. The brain will undergo reorganization after sensory deprivation. This plastic reorganization develops over time. Critical periods and the influence of the duration of sensory deprivation affects the reorganization of sensory cortices [9]. Hearing loss impacts brain function. The brains of people with different degrees of hearing loss respond differently. This is due to cortical reorganization [10]. It is the brain's tendency to adapt to a loss by rewiring itself [11].
Comprehension includes the selection of the correct meaning [12] to the words according to the context, and the ability to grasp, the meaning of a larger unitary idea. Understanding the meaning of the word is word level comprehension. It is also called lexical comprehension. Syntactic and vocabulary skills has often been suggested to be a fundamental factor in the progress of reading in the deaf population [13] [14]. Several studies have used Electroencephalography (EEG) as a physiological measure to determine the comprehension. EEG is a measure of the electrical activity of brain signals or electrical pulses at various locations that are picked up by electrodes connected to the scalp. It is a complementary source of feedback as non-invasively [15] feasible from as close to the brain. EEG offers good time [16] and spatial resolution [17].The speed of the brainwave is measured in Hertz (cycles per second) and they are separated into various bands [18]. The EEG signal is a voltage signal that can be calculated on the scalp surface coordinated neural activity from large areas (neurons groups firing at the same pace) manifested as synchronisation.
[20]. Mazaheri et al. investigated theta, alpha and beta frequency oscillatory behaviour, although previous studies have included these bands in different aspects of language processing, including lexical and semantic processing. [21]. Most of the low-frequency (infra-slow oscillations below0.5Hz) EEG studies have concentrated on different forms of cognitive tasks. [22]. Infra-slow oscillations are beyond the traditional clinical EEG bandwidth, but with the advent of digital signal processing, they have recently found clinical relevance. Studying the kinds of cognitive processes in each of the EEG bands that modulate behaviour allows one to understand the meaning of the connections that we observe between the behaviour of the EEG and our tasks. In addition, it helps us to identify the appropriate features to be used in training classifiers and thus to assess the EEG.
Event-related brain potential (ERP) estimate activity [23] which is harmonized to certain external stimuli, such as reading a word, or a sentence. The signal from an ERP is poor (5-10 μV) and noisy EEG signal is stronger (50-100 µV) [24]. So the actual signal is extracted from the noisy signal by an averaging procedure. This is accomplished by time-locking multiple EEG recordings called trials and averaging. Thus the event-related brain activity is got. Time domain refers to the analysis of physical signals with respect to time.
The spectral power and spectral entropy characteristics of gamma rhythms were extracted from the reported Auditory Evoked Potential (AEP) signals in a study conducted in Malaysia and applied to machine-learning algorithms to categorise the AEP signal dynamics in terms of specificity, sensitivity and accuracy of the left and right ear classification in 9 subjects, and observed as 96.75 per cent accuracy in separating the typically hearing and deaf subjects [25].
Recent studies with individuals who are deaf used machine learning to extract information by classification [26][27] by classifiers [28] [29]. One of the key theories in machine learning is that training data on which the classifier is trained and test data on which the classifier is evaluated are part of the same feature space [30]. The performance of the classifier is measured in terms of recognition rate [31]. Support Vector Machines [32], K Nearest Neighbor [31], ensemble averaging, are the classifiers used here. Feature extraction and classification are two critical areas in analysis [33].

Study Design and Method
Figure1 depicts path way of word level comprehension recognition system in deaf adults using the EEG signals. In the present work, the EEG data is acquired from the deaf adults who are graduates from National Institute of Speech & Hearing (NISH), Kerala using Clarity Brain Tech devices, when they are comprehending progressive words in English, ISL and Malayalam. The acquired raw EEG data is subjected to various noises due to the environmental factors and the movement of the participants. Using different digital filter, these artifacts are eliminated by pre-processing of the raw EEG signal. Statistical, linear and nonlinear characteristics were derived by DWT from the filtered EEG signal. Using machine learning algorithms such as Ensemble, SVM, and KNN classifiers, the comprehension of words in three languages (classes) for both group DAC and DAA were categorised.  (2) and Sign Language (1).

Design of EEG acquisition Protocol.
As the language of deaf adults with no early intervention is not appropriate to their age, words used to give stimulus were chosen from the words found in their note books during graduation study. This was to make sure that they were exposed or taught those words, so that they can comprehend during EEG recording. Three sets of twenty seven words were chosen. Each of the twenty seven words belong to nine sets of three level progression words (Example: Mother, Grandmother, Great grandmother; Stone, Rock, Mountain) which were displayed in the form of PowerPoint presentation. Three sets were in English, three in Malayalam and three in ISL for each trial.
The experiment protocol for each participant was three trials with 15 seconds interstimulus interval (ISI). Each trial lasts for duration of 135 seconds. Each word appeared on the screen for 5 seconds.
Recording started with a baseline period of 15 seconds followed by three trials of nine sets of words, with ISI as shown in Figure 2. The participants were then given an assessment questionnaire to check the following. 1. Number of words comprehended 2. Whether they identified the relationship. If they did not recognise, hints were given and then checked.
All participants read the procedure and instructions of EEG recording prior to recording. EEG recorded while the participants were reading and comprehending the words displayed as a PowerPoint presentation. After EEG recording, participants were given a questionnaire which checked their comprehension. The participants were seated comfortably in a chair and the electrodes were fixed on the scalp. The EEG was sampled with 18 electrodes of Clarity Brain Tech EEG device with 32 channels. Data with a sampling rate of 256 Hz and 10-bit resolution was recorded. Stimuli were progressive words in English, Malayalam and Indian Sign Language (ISL). These were presented as Power point presentation in the centre of a 17" laptop kept approximately 2.5 feet from the participants. Word stimuli were presented in three trials with interstimulus interval as in Figure 2. Each word was presented once during the experiment, for a total of 27 English words, 27 Malayalam words and 27 words displayed using ISL videos. EEG electrodes were placed on the scalp following standard 10-20 International placements [34] and the electrodes were bipolar with channel and a reference point. One ECG electrode was fixed on the hand. While recording, impedance was maintained below 10 kΩ at all electrodes.
Using a longitudinal bipolar montage, all channels were re-referenced, which improves the signal by removing similarities in neighbouring channels, leading to magnification of the characteristics associated with that area and elimination of common noise between channels. Bipolar mounting is a simple and fast spatial philtre widely used to decrease the background noise in the EEG to increase the signal-to -noise ratio. Positioning of electrode is referred as montage [33]. Eighteen EEG channels throughout the cerebrum were used to obtain eighteen bipolar channels, viz. The participants were given their own time to settle down and they were presented the PowerPoint presentation as shown in Figure 3. The comprehension EEG signals were collected while participants were reading the words. Eighteen electrodes were used to acquire EEG waves and one was connected in the arm to record ECG waves. The signal measurement system consists of electrode signals as input, which was integrated together after some delay. These were first amplified here, using differential amplifier. The disparity between the two inputs are magnified. It will subtract an unnecessary signal common to the two inputs. Then filtered and converted to digital output. Here the electrodes used were bipolar with the channel and a reference point, the difference of activities at those points are recorded as EEG.
All participants read the procedure and instructions of EEG recording prior to recording. It was also interpreted in ISL. A video of EEG recording procedure was also shown to each participant. EEG was recorded while the participants were reading and comprehending the words displayed as a PowerPoint presentation. After EEG recording, participants were given a questionnaire that checked if they have comprehended the words correctly. This questionnaire was also interpreted in ISL. There were a total of 81 words (Malayalam=27, English=27, ISL=27) displayed as stimuli. DAA participants decoded 80 and 79 words respectively, whereas DAC participants decoded 60 and 64 words each. DAC confessed that they comprehended all the ISL words and found difficulty with Malayalam and English words. Whereas, though the DAA comprehended almost all words, they found both English and Malayalam words to be easy, they said. Number of words comprehended is shown in Figure 4.

Pre-Processing
EEG data obtained from scalp electrodes can be viewed as a neural activity description, and objects are independent of each other. The raw EEG signals that was acquired from the participants were corrupted with a lot of physiological and non-physiological artifacts. The artifacts mainly consist of Power line Interferences (PLI), Eye blink, Eye movement, muscle artifacts like yawning, swallowing, laughing, stretching, mouse clicking etc.
Artifacts introduce spikes and this can affect the useful information from the EEG signal. Artifact can mimic almost any type of electrocerebral activity on the EEG [35]. To ensure proper examination and diagnosis, these undesirable signals must be extracted or attenuated from the EEG [36]. These artifacts are removed using Butterworth filter [37], FIR filter [38], IIR filter [39], Chebyshev filter [40], Adaptive filter [41]and Wavelet filter using discrete wavelet transform (DWT) [42].
In this work the acquired EEG signals were filtered and pre-processed using Moving average Low pass filter of window size 13 with cut off frequency 30 Hz and FIR filter of kaiser window parameter 0.37 and pass band frequency of 0.1 Hz. Filtered data comprises of the bands namely infra-slow oscillations, delta, theta, alpha, beta which has useful relevant information. Signals were sampled at the rate of 256Hz.

EEG Feature Extraction
MATLAB software was used for feature extraction and classification. Three parameters are included in the MATLAB programme: frequency, amplitude and duration, and stimulus signals are generated accordingly [40]. Since the Daubechies wavelet reflects the EEG signal characteristics, mathematical, linear and nonlinear characteristics were extracted using the EEG signal amplitude DWT (dB8) [43]. These wavelet functions are best suited for the stationery and non-stationary signals. Wavelets are the small oscillatory waves which can be used for the multiresolution analysis of the signal. Based on its similarity to the mother wavelet function characteristics of wavelet coefficients in the LF and HF bands, the Daubechies wavelet is selected. A single prototype feature called the mother wavelet is used, using the DWT to decompose the input signal based on scaling and shifting parameters. The mother w avelet function Ψa, b (t) is given as where a, b ϵ R, a > 0, and R is the wavelet space. Since the option of a prototype function as the mother wavelet should at all times fulfil the permissibility condition, the scaling factor and the shifting factor are the parameters 'a 'and' b '. (Equation 2), Where the fourier transform of Ψ (ω) is Ψ (a, b).
The features were calculated as in equations (3) to (12). [ Variability indicates how much spread out a set of data is. Hurst The characteristics of delta theta and alpha bands were extracted from the pre-processed signals. The time domain linear statistical features (Frequency, Mean, Maximum, Minimum, Skewness, Standard Deviation, Variance, Variability, Kurtosis) and non-linear features (Hurst, Approximate entropy) were extracted.

Classification on the basis of onset of deafness
The features that were extracted using DWT were statistically analysed using SPSS tool one-way analysis of variance (ANOVA). A Descriptive analysis was done by comparing the means of the derived features. The features whose p-value were less than 0.05 were chosen to be more significant for the analysis and are used for further processing. We selected significant features and classified them using the KNN, SVM and Ensemble classifier. Input variables (also known as predictors, features, or attributes) are based on a k-nearest neighbour classification model and output (response) is returned by 'fitcknn'(MATLAB function). The k value is chosen from 1 to 10 since the number of classes is two for classification. The distance metric that is used in KNN before any decision is the Euclidean distance [46]. SVM classifier produces an optimal. SVM is a linear machine that embeds the input data (features)which is in ndimensional space into high k-dimensional space generally k>n by nonlinear mapping of (x). A better generalization is done by optimising the margin while allowing some of the training set misclassification to prevent over fitting. [47]. Ensemble classifiers is a collection of classifiers whose individual choices are combined in some way (typically by weighted or unweighted voting) to classify new examples. The IOP Publishing doi:10.1088/1757-899X/1070/1/012083 9 study of methods for building good ensembles of classifiers [48] has been one of the most successful areas of supervised learning science.
In this work hold out validation was done to evaluate the performance of the classifiers. 70 percent and 30 percent of the dataset was used as training and testing sets for classifying two onset of deafness. % = * 100

Preprocessing
As shown in Figure 5 the raw EEG signal has artifacst such eye blink, muscle movement, power line interference, and other high frequency and low frequency noises. These artifatcs were removed using Moving average Low pass filter of window size 13 with cut off frequency 30 Hz and FIR filter of aiser window parameter 0.37 and pass band frequency of 0.1 Hz, and given as figure 6.

Raw EEG
Eighteen channels were connected. In this plot of raw signal, the channels FP1-

Statistical Data Analysis of the extracted features
Using one-way ANOVA the statistical significance of all twelve features obtained using DWT from the EEG signal was studied. The p-values are marked in Table 3. The results brought by this study also rise some interesting issues that justify additional research. While comparing the three languages, Maximum, Skewness, Approximate Entropy and Hurst showed significant values whose p-values were less than 0.05 for DAA. It is also noted that for DAC, the features Variability and Approximate Entropy gave significant values.

Classification of Congenital and Acquired deafness
The classification accuracy(figure 7) is shown in Table 4. Classifier SVM gives the maximum accuracy for all features analysed as well as the best feature identified which is Entropy. Using Entropy feature,

Conclusion
The purpose of this study was to explore, through the use of a technological tool, the word level comprehension in the process of reading Malayalam, English and ISL words. Artifacts introduce spikes, which can be confused as rhythms in neurology. To ensure proper examination and diagnosis, these undesirable signals must be extracted or attenuated from the EEG. The PLI is introduced whenever there is an electrical interference from other devices. The acquired EEG was filtered to remove artifacts and 12 features were extracted. Approximate entropy gave the best value. Classification was done for that using SVM classifier gave the highest accuracy. For DAC, ISL words seem to be the best comprehended ones. However for DAA, both ISL and English are comprehended equally. Words used to give stimulus were chosen from the words found in their note books during graduation study. This was to make sure that they were exposed or taught those words, so that they can comprehend during EEG recording. This may be the reason for English to get equal accuracy as ISL in DAA.

Limitations and Scope for Future Studies
This study compares electrophysiological signals of English, Malayalam and Indian Sign Language. The limitations are less number participants and less recording time. There is a scope for future studies on the features and classifiers for comprehension with more participants.