extfieldsofstudy:
Engineering
Computer Science
Physics

text:
Dense Recurrent Neural Networks for Inverse Problems: History-Cognizant Unrolling of Optimization Algorithms

Inverse problems in medical imaging applications incorporate domain-specific knowledge about the forward encoding operator in a regularized reconstruction framework. Recently physics-driven deep learning (DL) methods have been proposed to use neural networks for data-driven regularization. These methods unroll iterative optimization algorithms to solve the inverse problem objective function, by alternating between domain-specific data consistency and data-driven regularization via neural networks. The whole unrolled network is then trained end-to-end to learn the parameters of the network. Due to simplicity of data consistency updates with gradient descent steps, proximal gradient descent (PGD) is a common approach to unroll physics-driven DL reconstruction methods. However, PGD methods have slow convergence rates, necessitating a higher number of unrolled iterations, leading to memory issues in training and slower reconstruction times in testing. Inspired by efficient variants of PGD methods that use a history of the previous iterates, we propose a history-cognizant unrolling of the optimization algorithm with dense connections across iterations for improved performance. In our approach, the gradient descent steps are calculated at a trainable combination of the outputs of all the previous regularization units. We also apply this idea to unrolling variable splitting methods with quadratic relaxation. Our results in reconstruction of the fastMRI knee dataset show that the proposed history-cognizant approach reduces residual aliasing artifacts compared to its conventional unrolled counterpart without requiring extra computational power or increasing reconstruction time.

Subsequently, iterative algorithms are employed to solve the regularized inverse problem by alternating between enforcing physics-driven data consistency and regularization using preselected priors.
Recently, deep learning (DL) has emerged as an alternative state-of-the art technique for solving such inverse problems in imaging applications . Several of the earlier works concentrated on training convolutional neural networks (CNN) to solve the problem without incorporating explicit knowledge of the acquisition physics, typically requiring re-training when acquisition parameters are modified [18][19][20][21][22][23][24][25][26][27][28][29][30]. Another line of work focuses on a physics-driven formulation, where the known forward model is incorporated into training, utilizing domain knowledge to solve the corresponding inverse problem [31][32][33][34][35][36][37][38][39][40][41][42][43][44][45]51]. In physics-driven approaches, a conventional iterative optimization problem, which lands itself to a regularized least-squares formulation for modalities such as MRI and CT, is unrolled for a fixed number of iterations. This unrolled network alternates between data consistency and regularization steps similar to the conventional algorithm, where the regularizers are implemented via neural networks. The unrolled network is trained end-to-end to learn the network parameters, which predominantly characterize the regularization units.
In this study, we seek to develop a new methodology for unrolling inverse problem optimization algorithms into dense recurrent neural networks (RNN) for improved reconstruction performance. Motivated by accelerated PGD approaches, Fig. 1: (a) The conventional unrolling for iterative algorithms for inverse problems and (b) the proposed history-cognizant unrolling leading to a dense recurrent neural network architecture. In both cases, the algorithms are unrolled for T iterations, where each unrolled iteration consists of proximal (or regularization) unit (P), and a data consistency unit (DC). The P unit is implemented implicitly via a neural network, while the DC unit corresponds to a gradient descent step for the proximal gradient descent algorithms and typically to conjugate gradient for variable splitting methods. For the proposed history-cognizant unrolling, an additional 1 × 1 convolutional layer combines all previous proximal operator outputs prior to the DC unit, where C denotes concatenation.
which calculate the gradient descent at a combined history of previous iterations rather than the immediate previous iteration only [52][53][54][55][56][57][58], we propose a history-cognizant unrolling of the optimization algorithms with skip connections across unrolled iterations, leading to a dense RNN architecture. Skip connections have been previously shown to facilitate information flow through neural networks in architectures such as ResNet [59] and DenseNet [60]. Different than these previous neural network design problems, we propose to utilize such connections in an unrolled RNN architecture for solving inverse problems, which readily comes with its domainspecific design considerations. We compare the conventional unrolling and the proposed history-cognizant unrolling of the PGD algorithm in accelerated multi-coil MRI reconstruction using the fastMRI knee dataset [61], showing both visual and quantitative improvement in reconstruction quality with the proposed methodology. We also extend the same unrolling strategy to other optimization algorithms and investigate the performance improvement.

A. Forward Model and Inverse Problem
Let y denote a vector of noisy measurements from an imaging system and x be the corresponding underlying image. The forward model for this system is generally given as y = A(x) + n, where A is called a forward operator and n is measurement noise. In a number of medical imaging scenarios, such as MRI and CT reconstruction, the forward model is linear, leading to a simplified model where A is a linear forward operator and n is measurement noise, which will be the focus of the rest of the study. The corresponding inverse problem is typically ill-conditioned. Thus, regularization is frequently used, leading to a regularized least-squares method: arg min where the first term enforces consistency with measurement data and R(·) denotes a regularizer. In classical reconstruction approaches, regularizers have included Tikhonov [1,2], edgepreserving functions [62] or sparsity-promoting terms in predefined domains [3,4,6,7]. A variety of algorithms can iteratively solve the optimization problem in (2) such as PGD [63], VS [64][65][66] and PD [67,68] methods.

B. Physics-Based DL Reconstruction by Unrolling PGD
Several physics-based DL methods have unrolled a PGDbased method [32][33][34][35][36] for a fixed number of iterations. Such PGD-based unrolling alternates between a proximal operation that is implicitly defined by a learned neural network, and a gradient descent step to enforce data consistency, as follows: where µ i is the gradient descent step size at unrolled iteration i, and the proximal operation in (3a) is performed by a neural network. This leads to the schematic in Figure 1a, where the unrolled RNN consists of multiple blocks of proximal (P) and data consistency (DC) units. The former is a neural network and the latter implements a gradient descent step based on the known forward encoding operator. The unrolled RNN is then trained end-to-end to learn the mapping from the acquired sensor-domain data to reference images for medical imaging reconstruction.

C. Proposed History-Cognizant PGD Unrolling with Dense Recurrent Neural Networks
One of the major appeals of the PGD unrolling scheme is its computational simplicity. However, PGD algorithm itself has a slow convergence rate [62], potentially requiring a large number of iterations (i.e., a large number of blocks in Figure  1a) when unrolled into a neural network for sufficient reconstruction quality. In turn, this increases memory requirements and impedes training complexity. On the other hand, several methods have been proposed to accelerate convergence rate of PGD [52][53][54][55][56][57][58]. The underlying theme for these methods is to calculate the gradient step not only at the most recent proximal operation output, but at a linear combination of all past such outputs.
We propose to adapt this general accelerated PGD approach for network unrolling. In this setting, the accelerated PGD methods involve the following steps: where F(·) forms a linear combination of the previous proximal operator outputs. As a special case, Nesterov's method uses the previous two outputs [53], while more general versions have also been studied, in which all the past iterations are used [54,58]. For our unrolled network, we propose to learn F(·) during training instead of specifying a pre-determined form for the linear combination of previous iterates F(·) as in Nesterov's method [53]. This history-cognizant unrolling of the accelerated PGD algorithms leads to a dense recurrent neural network architecture (Dense-RNN) with skip connections across all unrolled iterations, as depicted in Figure 1b. In addition to the P and DC units to perform proximal operation (or regularization) and data consistency updates, each unrolled iteration also consists of a 1 × 1 convolutional layer implementing F(·) on the concatenation (denoted by C) of all past proximal operator outputs prior to the DC unit. We refer to this method as history-cognizant PGD (HC-PGD) in the rest of this study.

III. MULTI-COIL MRI RECONSTRUCTION EXPERIMENTS
AND RESULTS The proposed history-cognizant algorithm unrolling approach was compared to conventional algorithm unrolling in multi-coil MRI reconstruction. In a multi-coil MRI system with n c coils, y is the sub-sampled noisy data from all coils and x is the underlying image to be recovered. The forward linear operator A : C M ×N → C P in this system includes sensitives of the receiver coil array and a partial Fourier matrix for sub-sampling in k-space, i.e. spatial Fourier domain [1].

A. Knee MRI Datasets
Knee MRI datasets were obtained from the New York University (NYU) fastMRI initiative database [61] to investigate accelerated MRI reconstruction using conventional unrolled RNN and Dense-RNN architectures. Data were acquired without any acceleration on a clinical 3T system (Magnetom Skyra, Siemens, Germany) with a 15-channel knee coil. Data acquisition was performed using 2D turbo spin-echo sequences in coronal orientation with proton-density (Coronal PD) and proton-density weighted with fat suppression (Coronal PD-FS) weightings. Relevant imaging parameters were: resolution = 0.49 × 0.44mm 2 , slice thickness = 3mm , matrix size = 320 × 368 for both datasets.
All data were uniformly sub-sampled at an acceleration rate of 4 by keeping the central 24 lines using the mask provided in the fastMRI database [61] retrospectively. Uniform undersampling for 2D acquisitions is typically considered a more challenging setup for DL reconstruction than random undersampling [31]. A 24 × 24 central window was used to generate coil sensitivity maps using ESPIRiT [69]. The training sets consisted of 300 slices from 15 subjects for each sequence. Testing was performed on 10 different subjects for both sequences.

B. Implementation Details
The optimization algorithms were unrolled for T = 10 iterations. The proposed HC-PGD algorithm was implemented using the Dense-RNN architecture in Figure 1b. Proximal operator unit outputs are first concatenated and then combined using convolutional layers with a 1 × 1 kernel size and 2 output channels (real and imaginary components) before being inputted to the subsequent data consistency unit. On the other hand, the PGD algorithm was implemented using a conventional architecture that only includes P and DC units, without skip connections, concatenation of previous iterates or 1 × 1 convolutions.
The same CNN was used to implement the proximal operation for both PGD and HC-PGD. A residual network (ResNet) that has been established in a different regression problem was utilized [70]. This CNN consisted of 15 sub-residual blocks each having two convolutional layers of kernel size = 3×3 and output channels = 64. The first and second convolutional layers in each sub-residual block are followed by a ReLU activation function and a scaling factor of 0.1, respectively. In addition to sub-residual blocks, two input and output convolutional layers without any nonlinear activation function are to match number of desired channels. For both algorithm unrolling schemes, the proximal units shared parameters across unrolled iterations [39], leading to a total of 592,138 and 592,358 parameters for PGD and HC-PGD, respectively.
All networks were trained end-to-end, as summarized in Figure 2, by using Adam optimizer to minimize where x i ref , y i and A i are the reference SENSE-1 image, undersampled data and forward encoding operator for slice i, respectively and N is the number of training slices. f (y i , A i ; θ) denotes the network output for slice i with the corresponding measurements and forward encoding operator, while θ contains the trainable parameters of the networks. A normalized 1 − 2 loss [49] was used for L(·, ·): with u and v representing reference fully-sampled and network output images, respectively. Training was performed over 100 epochs. All algorithms were implemented using TensorFlow, and processed on a workstation with an Intel E5-2640V3 CPU (2.6GHz and 256GB memory), and two NVIDIA Tesla V100 GPUs with 32GB memory.

C. Performance of HC-PGD versus PGD Unrolling
Experiments were conducted on both Coronal PD and PD-FS datasets to compare the reconstruction quality of the conventional PGD unrolling and the proposed HC-PGD unrolling with the Dense-RNN architecture. A total of 380 slices from 10 subjects (excluded from training) were utilized for testing. Reconstructed testing data were evaluated quantitatively using normalized mean square error (NMSE) and structural similarity index (SSIM) metrics with respect to the fully-sampled reference image. Figure 3 depicts a representative slice from the Coronal PD dataset reconstructed using DL-MRI with PGD and HC-PGD unrolling methods. The first row shows the reference fully-sampled image, as well as the zero-filled acquired image, which serves as the input x (0) to the network. Difference images with respect to the reference image, scaled by a factor of 10, are also provided to facilitate comparison by showing errors. DL-MRI based on the proposed HC-PGD unrolling was able to suppress the strong aliasing artifacts successfully, while residual aliasing is visible on the DL-MRI reconstructions based on conventional PGD unrolling. For both methods, there is a slight degree of blurring in the final images, consistent with previous reports on DL-MRI methods with similar training loss functions [31]. Quantitative evaluations, shown in the lower left corner of the images, also confirm the improvement using the proposed history-cognizant unrolling with Dense-RNN architecture compared to the conventional algorithm unrolling.
The reconstructed images for a representative slice from the coronal PD-FS dataset in addition to the difference images scaled by a factor of 10, as well as the reference and zero-filled images are shown in Figure 4. There are slight residual aliasing artifacts in the DL-MRI reconstruction with PGD unrolling. The proposed HC-PGD unrolling successfully removes these artifacts further. The quantitative metrics also show improvement for the reconstruction results in moving from PGD to HC-PGD unrolling. Figure 5 summarizes the quantitative analysis of the DL-MRI reconstructions using NMSE and SSIM metrics for the PGD and HC-PGD unrollings for both the coronal PD and PD-FS datasets. The box plots depict the interquartile range with the center at the median of the metric for the corresponding reconstruction methods. Both metrics are visibly improved over the whole test sets when using the proposed history-cognizant algorithm unrolling over the conventional unrolling approach. Further statistical tests were performed using the Wilcoxon signed rank test with a significance level of P < 0.05, indicating that all differences are statistically significant based on both metrics and in both datasets.

D. Insensitivity of Dense-RNN Improvement to the Regularization CNN Choice
In order to verify that the improvements between PGD and HC-PGD unrolling are not limited to the choice of CNN used Fig. 3: A representative slice from the Coronal PD dataset with 4-fold acceleration rate using a uniform undersampling pattern, reconstructed with DL-MRI reconstruction based on the conventional PGD and the proposed HC-PGD unrolling, as well as error images with respect to the reference fully-sampled image scaled by a factor of 10. Zerofilled image corresponds to the input to the network, x (0) . The green rectangle in the reference image mark the zoom-up area shown in the second and last rows for each method. Residual aliasing artifacts, which are visible in the PGD unrolling results, have been suppressed using the HC-PGD scheme. Improvements in the quantitative metrics, shown on the lower-left corner of the images, also align with these visual observations. for the proximal operation, the two approaches in Figure 1 were also implemented using a U-Net architecture for the regularization CNN. This was based on the U-net used in Fig. 4: A representative slice from the Coronal PD-FS dataset with a 4-fold acceleration rate using a uniform undersampling pattern, reconstructed with conventional PGD and the proposed HC-PGD methods, along with the error images scaled by a factor of 10. The green rectangle in the reference image mark the zoom-up area. Proposed HC-PGD unrolling successfully removes some residual artifacts that are visible in the PGD. The quantitative metrics align with this observation. [61], which includes contracting and expanding paths up to 256 channels without batch normalization. Figure 6 depicts the results of this experiment on the coronal PD datasets, where the proposed history-cognizant unrolling outperforms the conventional unrolling methodology both visually and quantitatively. This highlights that the improvement from the

E. Performance Evaluation for Random Undersampling
In order to evaluate performance of the proposed historycognizant unrolling approach in random undersampling, we utilized an observation from a previous study [31,71], showing that physics-based networks trained on uniformly undersampled data can be transferred to random undersampling patterns at the same rate directly, even though the opposite is not necessarily true. Thus, we employed the networks trained on uniformly undersampled data, as detailed in Section III-B, to reconstruct test slices which were randomly undersampled at a similar 4-fold rate. The corresponding results, shown in Figure 7 indicate that the improvement trend remains similar to the uniform sampling scenario with the proposed HC-PGD method outperforming the conventional PGD in suppressing residual artifacts as observed in both Coronal PD and PD-FS datasets, which also align with the quantitative evaluation metrics in the images.

F. Extensions to Other Optimization Algorithms
While PGD unrolling is commonly used for DL-MRI reconstruction [32][33][34][35][36], alternative algorithms have also been utilized [37][38][39][40][41][42]. One such method for solving the objective function in Equation (2) is VS with quadratic penalty, which decouples data consistency and regularization terms into two blocks of variables [62] by introducing auxiliary variables to formulate a constrained objective function. This is then relaxed to an unconstrained problem by enforcing similarity between these variables via a quadratic penalty: arg min where z is an auxiliary variable and β is the quadratic penalty parameter. When this algorithm is unrolled as a neural network [38][39][40][41][42], reconstruction alternates between a proximal operation that is again learned by a neural network and data consistency, similar to PGD unrolling, as follows: where regularization update in (8a) is performed by a neural network and β was absorbed into the regularizer for notational convenience. The data consistency update in (8b) is computationally challenging in many practical scenarios, such as multicoil MRI reconstruction. Thus, this data consistency update can be solved iteratively via conjugate gradient (CG) to avoid matrix inversion [39].
As an extension to the HC-PGD approach, VS with quadratic relaxation can also be unrolled in a history-cognizant manner using the Dense-RNN in Figure 1, building on the optimization approaches in [56,57]. This will be referred to as the history-cognizant VS method (HC-VS). We note that the main difference to HC-PGD is that the DC units in HC-VS now implement an unrolled CG algorithm for a different least squares problem instead of a gradient step. To test the applicability of the history-cognizant unrolling scheme to different baseline algorithms, both VS and HC-VS were also implemented on the coronal PD and PD-FS datasets, using the setup described in Section III-B. In each data consistency unit, CG was implemented with 10 internal iterations.

IV. DISCUSSION
In this study, we proposed a history-cognizant method for unrolling optimization problems to solve regularized inverse problems in medical imaging using physics-based DL reconstruction. Our unrolling method builds on optimization approaches that utilize a history of the previous iterates via their linear combinations in updating the current estimate [52][53][54][55][56][57]. While these optimization procedures have been traditionally proposed to improve convergence rates or provide stronger theoretical guarantees, in the context of algorithm unrolling for physics-based DL reconstruction, they are used to enhance reconstruction quality without incurring additional computational complexity. Our method was pre-dominantly motivated by accelerated PGD approaches [52][53][54][55] and was shown to improve upon conventional PGD unrolling for multi-coil MRI reconstruction for multiple scenarios. We also explored how the unrolling can be applied to VS with quadratic relaxation for solving a regularized least squares problem, and showed that it can improve over the conventional unrolling approach.
The proposed history-cognizant unrolling methodology leads to several skip connections across iterations, leading to a densely-connected unrolled neural network. Dense neural networks have been shown to improve training by facilitating information flow through the network in forward propagation and by relieving issues concerned with gradient vanishing in backward propagation [60]. One of the advantages of these dense connections in the proposed unrolling is that the only computational overhead compared to conventional unrolling is the calculation of the 1 × 1 convolutions from the previous iterations. Thus, once trained, the computational time is increased <1.5% compared to the conventional counterpart, as depicted in Table I. One of the most commonly used accelerated PGD approaches is based on Nesterov acceleration [53]. In this case, the gradient descent is evaluated at a linear combination of the previous two iterates weighted by a step size µ and 1 − µ respectively. Theoretically, this approach achieves the optimal convergence rate for first-order methods [53]. As such, Nesterov-type unrolling for DL reconstruction was readily explored in [36] for coded-illumination microscopy, where the unrolled accelerated PGD network used the history of the two previous outputs of the proximal units. However, the main focus of this work was developing an illumination pattern design algorithm for a fixed reconstruction algorithm. On the other hand, when focusing on the reconstruction aspect for inverse problems, we found that the Nesterov-type unrolling was outperformed by the full history-cognizant Dense-RNN architecture in our initial experiments (not shown). Furthermore, the full history-cognizant unrolling subsumes Nesterov-type as a special case. Thus, we focused on this full history-cognizant case throughout our study to maximize the performance benefits.
In our experiments, the history-cognizant unrolling visibly improved the performance of PGD-based methods compared to its conventional counterpart for all datasets. In the VS setting, especially for the coronal PD datasets, the improvement is less pronounced. Coronal PD datasets have higher SNR compared to PD-FS datasets [31]. In this higher SNR case, the conventional unrolling of the VS algorithm readily provides a high-quality reconstruction. Thus, additional improvement with HC-VS is less apparent than in the PGD setting.
Our experiments in multi-coil MRI reconstruction pre-dominantly utilized uniform undersampling patterns. While random undersampling patterns are commonly used in literature for training and testing DL-based reconstruction, uniform undersampling remains the most clinically used approach, and was considered to be a harder problem compared to random undersampling in [31,71], as detailed in Section III-E.
To the best of our knowledge, the Dense-RNN architecture that arises from the proposed history-cognizant unrolling have not been used for regression problems in general, and inverse problems in particular. However, Dense-RNNs have been independently proposed for different problems, such as scene parsing or labeling [72] where semantic information of an image segment is to be extracted. In such applications, RNNs are used for each image unit to receive the dependencies from other units through recurrent information forwarding between adjacent units, which decays for further units. In this setting, Dense-RNNs improve labeling by capturing longrange semantic dependencies among image units, highlighting a similar memory dependence as in this study.

V. CONCLUSION
We proposed a history-cognizant approach to unroll a given optimization algorithm for solving inverse problems in medical imaging using physics-driven deep learning. The proposed approach leads to an unrolled dense recurrent neural network architecture. From a network-design perspective, this consequent architecture enhances training and performance by facilitating information flow through the network. Accelerated MRI reconstruction experiments show that the proposed unrolling approach may considerably enhance reconstruction quality without substantially modifying computational complexity.
ACKNOWLEDGMENT MRI data were obtained from the NYU fastMRI initiative database [61] which was acquired with the relevant institutional review board approvals as detailed in [61]. NYU fastMRI investigators provided data but did not participate in analysis or writing of this report. A listing of NYU fastMRI investigators, subject to updates, can be found at fastmri.med.nyu.edu.


Research on 4G And 5G Authentication Signaling

In recent years, the development and standardization of 5G networks have received extensive attention. 3GPP officially confirmed that Service-Based Architecture (SBA) is the unified infrastructure of the 5G core network (5GC). The Service-Based Interface (SBI) is an important part of the SBA for interaction between network function services within the 5GC. The fifth generation mobile network will support a range of use cases and requirements that are completely different from traditional 4G networks, such as providing a high level of authentication mechanisms. This paper is organized as follows: First, we compare the difference between 4G authentication and 5G authentication in three aspects including the 4G architecture, authentication process, and communication protocol stack, then implement the simulation environment, and finally evaluate the performance by comparing the delay.

Introduction
The rapid development of mobile Internet and IoT services has greatly promoted the advancement of mobile communication technology. 4G communication technology is gradually unable to meet people's growing demand for information transmission, and 5G technology is recognized by its reliability, timeliness, large bandwidth, and information transmission capability for large machines. 5G networks have greater advantages over 4G networks in terms of transmission rate, network performance, mobility, signal strength, and security. The 5G network has a higher density of connections, a faster experience, and lower air interface latency [1,2].
However, the diversified service scenarios of data transmission pose a serious challenge to 5G networks. Different business scenarios have different requirements for transmission security. Therefore, 5G designs should be compatible with a variety of security solutions. Closely related to security is the authentication in the network, so it is important to study the authentication process [3,4]. The main entity of 4G is shown in Figure 1. A number of modules with different functions are integrated into the 4G architecture to provide secure access and authentication. The modules related to authentication are as follows [5]:

4G System Architecture
The UE (User Equipment) consists of two parts, one is the UICC (Universal Integrated Circuit Card) and the other is the USIM (Universal Subscriber Identity Module), which is the basis of the USIM operation. The user's identity information and address information are stored in the USIM, such as the user's key. This information is shared with the AuC in the local area network, but the USIM copy cannot be saved in the AuC; the same is true for the IMSI (International Mobile Subscriber Identity). As the unique identifier of the user identity, IMSI can be divided into three main modules: MSIN (Mobile Subscription Information Number), MCC (Mobile Country Code), and MNC (Mobile Network Code).USIM plays an important role in the user authentication process.
E-UTRAN (Evolved Universal Radio Access Network) is a major component of the eNodeB (Evolved Node B). The eNodeB is an important bridge for the UE to connect to the network and is an essential component for connecting the core network.
The MME (Mobility Management Entity) is the primary control node of the network. The MME performs authentication, mainly responsible for the attach procedure, bearer processing (cooperating with the S-GW and P-GW), tracking the location of the UE and selecting a gateway (determining the routing of data packets).
Subscriber data and keys, including subscription profiles, are stored as the most important information for the user in the HSS (Home Subscriber Server). In short, the HSS is a database of user information.   Figure 2 shows the whole process of authentication.  Figure 3. S6a Protocol Stack Figure 3 shows the communication protocol stack between the MME and the HSS module. The interface used by the MME and HSS for communication is S6a, and the transmission protocol is Diameter. Diameter is one of the basic protocols, mainly used for authentication and accounting. The protocol must be used in conjunction with applications based on the protocol and provides a reliable transport mechanism, information processing and exception handling mechanisms similar to the TCP protocol [6]. The transport layer uses the SCTP (Stream Control Transmission Protocol) protocol. The Diameter message has a Diameter header and a series of AVPs (Attribute Value Pair).

5G System Architecture
There have been some changes in 5G authentication and 4G authentication. From the functional module, the HSS in 4G evolves into AUSF and UDM in 5G, and the MME of 4G evolves into AMF in 5G. The following is a detailed introduction of the functions of three network elements in 5G [7,8]： AMF (Core Access and Mobility Management) can implement many functions. For example, mobility management, registration management, verification of access identity and authorization mechanism implementation. UDM (Unified Data Management), which stores data that is equivalent to the HSS module in 4G, which is user data. Of course, UDM only stores a part of the data (user's subscription data), not all of the data. UDM also supports other features such as access authorization, user tag handling, authentication credential processing, and more. AUSF (Authentication Server Function): It mainly implements the authentication function in authentication, which is similar to the function implemented by the MME in 4G: generating a secret key, implementing two-way authentication for the UE, and supporting a consistent authentication model.  Figure 5 with Figure 2, the main changes are as follows: (1) The MME is divided into AMF AUSF, and the HSS becomes UDM; (2) Performing a hash calculation on RES and XRES; (3) Increased communication between AMF and AUSF (signaling 8-10). Figure 6. SBI Protocol Stack Due to fundamental changes in the core network architecture, the protocol stacks used in 4G core networks (such as Diameter / SCTP) are too heavy for flexible architectures. It may not meet the requirements of the 5GC. The SBA solution leverages the experience of web-based interaction design to provide flexible connectivity to services in distributed systems. Compared to a relatively fixed Web service, 5GC needs different architectures to support new services, which is more complex and more challenging. The SBA uses the SBI protocol stack as shown in Figure 6, the application layer uses the http2 protocol, and the message format is serialized in the json (JavaScript Object Notation) format [9,10].

Implementation
Two sets of source code were used in our study, one is openairinterface5g and the other is openair-cn. Openairinterface5g implements eNB and UE functions. Oaisim realizes system-level simulation, and oaisim includes simulation of LTE protocol stack as a whole. This code does not make any changes in 5G usage.
Openair-cn is an implementation of the 3GPP specifications concerning the Evolved Packet Core Networks which it contains the implementation of the following network elements: MME, HSS, S-GW, P-GW. For 5G, we made corresponding modifications on the basis of this code. It mainly involves the re-division of authentication network elements: AMF, AUSF, UDM. Three processes are re-opened in the source code, and the communication interface between the modules becomes SBI(http2+json).

Testing Cases
Firstly, some changes can be seen from Figure 2 and Figure 5 about the 4G and 5G authentication procedures, mainly because the change of NF leads to the change of the message flow. Secondly, it can be seen from Figure 3 and Figure 6 that 4G and 5G transport protocol stacks have changed (application layer, transport layer). These changes will bring about changes in performance, and an important index of performance is latency. Therefore, we have made certain measurements on the corresponding delays, including the following: (1) 4G REQ-RES: the time from the initiation of the request by the nas layer of the MME to the receipt of the response; (2) 5G REQ-RES: the time from the initiation of the AMF request to the receipt of the response.

Testing Environment
Table1 Hardware and software configurations The experimental system uses two multi-core servers, which are configured as shown in Table 1. One of them deployed openair-cn (core network) and the other deployed openairinterface5G (eNB and UE) to complete the deployment of 4G simulation system. We use the modified code to complete the deployment of the 5G simulation system.

Conclusion
Table2 Average latency As can be seen from Table 2, compared with 4G, the 5G authentication delay increased by 9273us, and the delay increased by approximately 6.46%. The SBA core network architecture introduced by 5G also brings a certain delay in the case of satisfying certain decoupling and meeting the requirements of the scenario, within the allowable range.


extfieldsofstudy:
Physics
Computer Science

s2fieldsofstudy:
Computer Science
Mathematics

text:
The basic set of test problems for ODE system solvers

Known mathematical packages, MATLAB, Maple, Mathematica, MathCAD and others may get wrong, often plausible, the result of numerical solution of ODE systems with low, given by default, the requirements for mathematical accuracy of the results of the numerical solution of ODE systems. Since ODE system parameters obtained usually experimental, with a low mathematical precision, so the requirements for the precision of the results of mathematical solutions of ODE systems is low (for example, in MATLAB package required precision is 0.001). The article offers a basic set of tests to assess the range of applicability of the relevant solvers. The basic set of test problems for ODE solvers systems include linear ODE systems with the known analytic solution and nonlinear systems with known graphics solution. Presented comparative results of solutions for proposed problems using MATLAB solvers and manzhuk program from a library of standard mathematical programs SADEL (Sets of Algebraic and Differential Equations solvers Library), which has been designed for reliable and accurate solving of systems of linear algebraic equations (LAE) and ODE systems. The results can be used in mathematical modeling of dynamic systems described by ODE systems.

Introduction
The programs for solving ODE systems use three classical problem statements for solving ODE systems: 1. The Cauchy normal form in the space of differential state variables (explicit form) [1]: ⁄ Where is the vector of the coordinate basis of differential variables of the state of dimension ; is the vector function of the right parts of the dimension ; is the independent variable (usually me . Initial conditions specified as , time of integration is , is the time of the beginning the integration, time of completion of integration. 2. Differential-algebraic form of different indices in the full coordinate basis (semi-explicit form) [2,3]: Where -vector of algebraic variables of the full coordinate basis of differential-algebraic variables of dimension , is the vector function of dimension . The agreed initial conditions are now specified as . This statement (1) of the task of ODE systems solutions are  2 widely used in the mathematical and numerical modeling automatic regulation and control systems, for example, MATLAB-SIMULINK [4], MVTU [5,6] and others. 3. The differential-algebraic form in the full coordinate basis, i.e. system of differential-algebraic equations (DAE) (implicit form): Wherevector function of dimension . The initial conditions are the same as in the previous case. Statement (2) is used in mathematical and numerical modeling of electronic and mechanical engineering systems, for example, the ODE systems solvers SPICE and ADAMS. The task (2) in the space of differential algebraic variables was set in the full coordinate basis by L. Petzold in 1982 and she developed the DASS (Differential Algebraic Systems Solver) program based on the Gear method with replacement of derivatives according to the Gear formula [7]:

Where
-Backward Differential Formula,integration step. The resulting system of nonlinear algebraic equations was solved by the Newton method at each integration step. The task of solving ODE systems in the extended space of differential-algebraic variables without transformations of the initial ODE systems is based on the statement (2) of the problem and has the form: (3) Here ⁄vector of derivatives of differential state variables with respect to time of dimension . Considering the fact that some matrix-vector transformations on a computer are not equivalent due to the limited bit grid of computers, the classical forms of representation of ODE systems should be used mainly for theoretical studies, provided that all transformations of the original DAE systems are performed analytically or in symbolic form. The stability domain of any explicit method is limited, therefore, implicit methods are recommended for solving hard problems [1,2]. But implicit methods also have their drawbacks. Using the implicit method for solving stiff and locally unstable problems may lead to qualitatively wrong solution [8]. It is undesirable to use implicit methods for modeling hybrid dynamic systems containing discrete and relay elements, since in this case the Jacobi matrix must be updated frequently. Finally, implementing implicit methods is much more complicated than explicit ones. Therefore, attempts to build explicit methods that are effective for hard problems do not stop [8]. This article discusses the basic set of test ODE systems, which includes both stiff and nonstiff ODE systems and comparing solvers of MATLAB with manzhuk program from the library of standard mathematical programs SADEL in this set of tests.

Methods for solving ODE-DAE systems
There are implemented 8 methods for solving ODE-DAE systems in the MATLAB. The choice of methods for solving ODE-DAE systems in the SADEL library is explained below. Since the explicit methods for solving systems of ODE and DAE are not universal and do not meet the requirements of the stability in the solution with variable step integration of the stiff ODE systems, then as basic integration methods for SADEL library were selected implicit methods of integration [2]. Implicit methods for solving ODE systems are reduced to iteratively solving nonlinear algebraic systems. In the software implementation of these methods, two main types of errors in the numerical solution of ODE systems should be distinguishedglobal error (or qualitative error) and local error (or quantitative error). In theoretical terms, the properties of ODE-DAE systems and methods for their solution are considered in relation to linear inhomogeneous ODE systems of the form: Where is a constant real matrix with size , is a known vector function of time. If the integration method is not suitable for solving the system (4), it is unsuitable for system (3) too. Denote the determinant of the matrix through . The spectrum of eigenvalues is the set of all eigenvalues of matrix , . In the general case is a complex value. It is known that if the eigenvalues are different, then the general solution of system (4) will be a linear combination of fundamental solutions and , is some constant. The analytical solution of system (4) can be represented as the sum of the individual fundamental solutions of a homogeneous system of ordinary differential equations with a matrix and the private solution of the heterogeneous system (4):  The classical control of the local integration error will ensure the accuracy of the solution, and to control the global integration error it is necessary to ensure the appropriate stability of the integration methods. To ensure obtaining a qualitatively correct solution for the above classes of problems, the integration method must be stable for stable ODE systems (I and II classes of tasks) and should be unstable for unstable ODE systems (IV and V classes of problems), therefore it is necessary that the integration methods be appropriately AL-stable (AL-stability -Absolute stability in Left half-plane of the complex stability plane of numerical integration), i.e. absolutely stable strictly in the left half-plane of the complex plane of stability of the integration methods [9] . With low requirements to the precision of solving ODE systems, this area of absolute stability guarantees the preservation of a qualitatively correct solution with increasing integration step in the plane , since with low integration accuracy the integration step can become so large that non-ALstable integration methods can obtain incorrect solution trajectories. Implicit one-step Runge-Kutta integration methods meet the AL-stability requirement [10]. Algorithms for the implementation of implicit one-step integration methods of Runge-Kutta type are based on the joint solution of systems of nonlinear algebraic equations for solved DAE system of the form (3) and systems of linear algebraic equations for integration methods , formed at the corresponding stages of the integration methods, relative to differential-algebraic variables , and derivatives of differential variables [11]. Were developed DAbc formulas (as the development of Ab T c Butcher methods [12]) for S-phasic implicit methods for solving DAE systems in an extended coordinate basis: Wherethe number of stages of the corresponding one-step method ; --th integration step; , , , method parameters. Based on these formulas, three integration methods were developed and implemented:  M1 -A-stable one-step implicit method of the first order of accuracy;  M2 -AL-stable one-step implicit method of the second order of accuracy;  M3 -AL-stable two-stage implicit method of the fourth order of accuracy. The main drawback of AL-stable methods is that they do not satisfy the requirements of Lstability and sometimes give "false fluctuations" in the solution, so these methods are not widely used in practice [2]. The solution to this problem is given in [10].

Test problems with a well-known analytical solution
It is proposed to develop 5 test linear homogeneous ODE systems 1) -5) with analytical solutions corresponding to the classes of tasks shown in Figure 1 as the basic set of test problems. Given the test linear system of ODE of the 2nd order:

⁄
Where is the vector of unknowns of system , is the matrix of constant coefficients of a system of size 2 2, is a constant vector.

Test problems with a graphical solution:
6) An example of the calculation of a tough second-order ODE system (MU is a stiffness parameter) -Van der Pol's test. Correct solution for this test shown in Figure 2.

7)
Example of electronic circuit simulation with a multi-period solution (5th order ODE system). Figure 3. Electronic circuit for test 7 The ODE system for this test is:   Waivers for problems with graphical solution are time of calculation more than 10 seconds or the wrong plot of the solution.

Test problems with a well-known analytical solution :
Failures for tasks with an analytical solution were recorded if the maximum relative error was more than 0.001 in a given integration interval. These tasks were used to test manzhuk solver and different MATLAB solvers. The final test results are shown in Table 1.

Test problems with a graphical solution
In view of the results published in [10,13,14], Table 2 shows the summary comparison solvers of stiff ODE systems of the aforementioned SADEL library, from well-known libraries and packages of mathematical programs. The real laser parameters manzhuk (М3 The sign (+) means the correct solution of the test using the method recommended for solving stiff ODE systems, with an accuracy . The sign (-) indicates an incorrect test solution using the method recommended for solving hard ODE systems with default parameters without warning the user and refusing to get the right solution. The sign (-+) means an incorrect test solution by the method recommended for solving hard ODE systems, with default parameters without warning the user and obtaining the correct solution by setting appropriate methods and integration parameters and increasing the calculation time. Sign (-++) means an incorrect test solution by the method recommended for solving hard ODE systems, with default parameters without warning the user and obtaining the correct solution by setting only the parameter specified accuracy of the method of integration and increasing the calculation time. NDno data. Figure 6 shows as an example the result of a qualitatively incorrect solution for the 1st variable of test 7) by the implicit method ode15s and correct solution by method M3 of program manzhuk. There were no warnings about the incorrect decision in the ode15s solver.

Conclusion
Most of the problems of solving systems of ordinary differential equations arising in practice cannot be solved analytically [15]. Numerical methods descend to the seventeenth centuryto Newton, who proposed using them to calculate the trajectories of comets. Manual calculations continued to be used until the 60s of the twentieth century, before the appearance of the first solver programs written in FORTRAN for the first computers [16]. Since that time, development of test problems for solving ODE systems has been carried out. Classical sets of test considered tests from books [1,2,17]. The first test comparison of the MATLAB ODE solvers is given in [18]. In MATLAB we can always choose the method and adjust its parameters to obtain qualitatively correct results, but most users are not experts in numerical methods, so the comparison should be made for the recommended default parameters of integration ODE solvers. For specific classes of problems to be solved, it is necessary to have sets of test practical and mathematical problems with a known deliberately reliable and accurate solution for the correct selection and adjustment of the corresponding solvers of ODE systems and the correct evaluation of the obtained results of numerical simulation. This article presents the results obtained in the continuation of the above works. Further studies focused on developing effective methods and algorithms for the automatic integration step selection for AL-stable implicit methods for the numerical solution of high-and ultra-high-dimensional ODE-DAE systems and testing these new solvers. The relevance and importance of these works is indicated in the [19,20].


Generic Multiview Visual Tracking

Recent progresses in visual tracking have greatly improved the tracking performance. However, challenges such as occlusion and view change remain obstacles in real world deployment. A natural solution to these challenges is to use multiple cameras with multiview inputs, though existing systems are mostly limited to specific targets (e.g. human), static cameras, and/or camera calibration. To break through these limitations, we propose a generic multiview tracking (GMT) framework that allows camera movement, while requiring neither specific object model nor camera calibration. A key innovation in our framework is a cross-camera trajectory prediction network (TPN), which implicitly and dynamically encodes camera geometric relations, and hence addresses missing target issues such as occlusion. Moreover, during tracking, we assemble information across different cameras to dynamically update a novel collaborative correlation filter (CCF), which is shared among cameras to achieve robustness against view change. The two components are integrated into a correlation filter tracking framework, where the features are trained offline using existing single view tracking datasets. For evaluation, we first contribute a new generic multiview tracking dataset (GMTD) with careful annotations, and then run experiments on GMTD and the PETS2009 datasets. On both datasets, the proposed GMT algorithm shows clear advantages over state-of-the-art ones.

Introduction
Visual object tracking is a fundamental problem in computer vision. Among different tracking tasks, we focus on generic (aka model-free) visual tracking, which requests little prior information about the tracking target and has been intensively researched due to its wide range of applications. Despite great advances in tracking algorithms, tracking in  real world is still challenging especially when target appearance is distorted or damaged due to view change or occlusion.
A natural way to alleviate the above issues is to use multiple cameras for tracking, which provides important multiview information for handling cross-view target appearance change and occlusion. Existing multiview tracking algorithms (Sec. 2.2), however, typically focus on specific targets like humans, and often rely heavily on detection or reidentification models. Another limitation is that cameras are often assumed to be static, so background subtraction and/or camera calibration can be used to facilitate target localization. These limitations largely restrict the generalization of multiview tracking algorithms to real world applications. Consequently, effectively using multiple cameras for generic visual tracking remains an open problem.
To address the above mentioned issues, we propose a novel generic multiview tracker (GMT) in this paper by encoding rich multiview information with learning-based strategies. A key innovation in GMT is a cross-camera trajectory prediction network (TPN), which takes tracking results from reliable views to predict those for unreliable ones. TPN effectively alleviates the problem caused by occlusion or serious target view change. Another novel component in our GMT is the collaborative correlation filter (CCF), which assembles cross-camera information to update a correlation filter shared among different views, and hence improves tracking robustness against view change. TPN and CCF are integrated into the correlation filter tracking framework, where the features are trained offline using existing single view tracking datasets for further improvement.
For evaluation, we first construct a new generic multiview tracking benchmark (GMTD) to address the scarcity of such benchmarks. Then, the proposed GMT algorithm is tested on this dataset and the PETS2009 dataset [13], and demonstrates clear advantages in comparison with state-ofthe-art tracking algorithms.
To summarize, we make the following contributions: • The learning-based generic multiview tracking framework, which requires little prior information about the tracking target, allows camera movement and requires no camera calibration. • The novel cross-view trajectory prediction network that encodes camera geometric relations for improving tracking robustness. • The collaborative correlation filter that learns an online cross-view model and hence achieves natural robustness against view change. • A new generic multiview tracking dataset with manual annotations per frame, which is expected to further facilitate research in related topics. The source code and the dataset will be released to public with the publication of this paper.

Generic Single View Visual Tracking
Visual object tracking is one of the most important tasks in computer vision and has attracted a great amount of research efforts. It is beyond the scope of this paper to review all previous work in tracking. In the following section, we choose to review some of the most relevant ones.
Our work is most related to correlation filter-based trackers. Based on Discriminative Correlation Filters (DCF), MOSSE is proposed in [4] to efficiently train the correlation filter by minimizing the sum of the squared error between ground-truth and output in Fourier domain. The idea is later adopted and extended in Kernel Correlation Filter tracking (KCF) [18] and since then starts to gain great amount of attention. Among many tracking algorithms along the line, the series proposed by Danelljan et al. [10,11,8,3] provide the main backbone to our study. In these studies, DSST [9] and fDSST [10] use multiple correlation filters to estimate object translation and scale separately. CCOT [11] further enhanced predictions by learning from multi-resolution feature maps. ECO [8] and ECO+ [3] make further improvements on feature representation, sample space management and online update scheme in order to obtain more intuitive tracker and win both accuracy and efficiency.
Another line of tracking algorithms, from which we borrow the initial feature representation, are the Siamese network-based trackers. Siamese Fully Convolutional networks (SiameseFC) [2] solves the problem by training a fully convolutional Siamese architecture to evaluate the semantic similarity between proposals and target image. Utilizing lightweight CNN network with correlation filters, DCFNet [35] performs real-time offline tracking. Recent extensions continuously improve the performance such as in [5,36,34,15].
Moreover, our work explicitly addresses occlusion and target view change, which have been studied explicitly as well in some single view tracking algorithms. One way to resolve the appearance variation of the target is maintaining effective sample sets [8,28,27], which involves balancing different aspects of the target, for correlation filters online-training. Another strategy is conducting complementary information, for example, [33] makes use of spatial information, [39] enrolls flow between frames as part of features, [17] add a semantic branch to enhance prediction. For occlusion, there are part-based correlation filter trackers like [6,7,25], which learn target appearance in parts and tend to be robuster to partial-occlusion. When targets have strong structure relation, like pedestrians in RPAC [26], KCFs are assigned to five different parts of each target for robustness. [28,27,12] set thresholds to evaluate the results of correlation filtering. Mei et al. [31] investigate sparse representation for occlusion detection. MUSTer [16] avoid drifting and stabilize tracking by aggregating image information using short-and long-term stores. Different from all above-mentioned methods, our study focuses on multiview visual tracking that takes input streams from multiple cameras simultaneously. While borrowing some components from these single view tracking algorithms, we develop novel strategies such as cross-view trajectory prediction and a multiview collaborative correlation filter. These strategies, as demonstrated in our carefully designed experiments, clearly improve the tracking robustness.

Multiview Visual Tracking
Multi-camera inputs have been used for visual tracking. For examples, Khan et al. [19] apply a planar homography constraint on calibrated cameras for tracking pedestrians on the ground, and show the power of multiview system with common overlaps; studies in [22,29] explore spatial relations of target object in multi-camera system by analyzing entry/exit rates across pairs of cameras; [1] exploit dynamical and geometrical constraints among static cameras in a linear model; [32] based on detection and re-identification methods for multi-target tracking; the multiview trackers [37,23,24,30] track a target in each view and match each instances between cameras that are intuitively capable of occlusion situations due to diversity of observation directions.

Problem Formulation
Given a set of synchronized video streams from different cameras/views, we aim to localize the target (initialized in the first frames) across time. More specifically, let the system input be I t = {I c t } nc c=1 at time t for n c cameras, and let B 1 = {b c 1 ∈ R 4 } nc c=1 be the initial target bounding boxes for all views. Then, our multiview tracking task is to localize the target by finding B t = {b c t ∈ R 4 } nc c=1 , given {I 1 , I 2 , . . . , I t } and B 1 , 1 where b c t is the target bounding box with four parameters in view c at time t. We also define a trajectory set for each view c and g c t is the center of b c t in a consecutive time period from time t 1 to time t 2 .

Framework Overview
The key motivation of our generic multiview tracker (GMT) is to explore rich cross-view information to improve tracking robustness, especially against occlusion and target/camera view change. We adopt the correlation filterbased tracking framework as the backbone, and equip it with the novel collaborative correlation filter (CCF) and cross-view trajectory prediction network (TPN) techniques. An overview of pipeline of GMT is given in Figure 2, and we briefly describe it as follows.
During online tracking, for newly arrived multiview images I t , GMT locates the target (i.e., calculates B t ) in three major steps. First, for each view c and scale k, a region of interest (RoI) patch U c k is prepared around b c t−1 . The patch is then fed into the feature extraction network φ(·), which is shared among different views, to generate feature maps, denoted by X c k , for each view c. Second, each feature map is convolved with the shared CCF f for initial target localization, producing confidence map Y c k . The maximum response over different scales k is then picked for the initial tracking results. For sufficiently confident results, they will be assembled cross view to update CCF. In other words, CCF is online updated to enhance its robustness against view and appearance change.
Third, for a view with low confident initial tracking results (e.g., view 3 in Figure 2), TPN will be used to estimate its tracking result by implicitly taking into account the geometric relations among different views/cameras.
The overall framework is summarized in algorithm 1. More details are described in the following subsections. 1 The estimated {B 2 , . . . , B t−1 } can be used as well.

Algorithm 1: Generic Multiview Tracking
Input : I t : input images at time t; G t0,t−1 : previous trajectories of all cameras; f : dynamically updated collaborative filters; B t−1 : tracking results (boxes) in last frame; Z c : training samples for each view c; Output: B t : tracking result for time t; G t0,t , f, Z c : updated results;

Spatial-aware Feature Extraction Network
In order to improve the feature adaptability to correlation filter-based multiview tracking, we fine tune feature extraction model based on ResNet-50 that is pretrained on Ima-geNet. The offline fine tuning involves two components: the feature extractor module φ(·) and the correlation filter module ϕ(·), as shown in Figure 3.
We construct the training set from existing single view tracking datasets including VOT2017 [20], OTB100 [38] and LaSOT [14]. Different frames in a sequence can be regarded as the multi-view appearances of the same object. Data augmentation is conducted by slightly disturbing object locations. We randomly choose 16 sample pairs from the same sequence as a training batch.
The training for each batch has two stages. At the first stage, the 10 training pairs are used to train the correlation  filter f train by minimizing the following objective function: where Y * is target score map; θ f is the parameters for feature extractor; η is a penalty parameter; and ψ cos (·) is the Hann window function. After the first training stage, we obtain the optimal filter f * train of this current object. Then, we use the rest of the sample pairs to train φ(·). The output response map may have some blurred noise which is supposed to be zeros when only L2 loss is used. So we add a gradient-like term to the objective function to alleviate this phenomenon. The objective function is: where ∇ s (·) is the Sobel operator. We use Adam optimizer to train our spatial-aware feature extraction networks. Note that f train is only used during offline training process.

Collaborative Correlation Filter
View change is a notorious issue that troubles single view trackers. Fortunately, in the multiview tracking setup, images captured from different cameras naturally provide cross-view information for building reliable tracking models. Therefore, we extend the traditional correlation filter to a collaborative one. Specifically, during tracking, we update the correlation filters online with information collaboratively collected from all sufficiently reliable views.
Denote the training samples dynamically collected from view c by Z c = {X c j } mc j=1 with m c samples. We train a shared multiview collaborative correlation filter f using all samples from different sample sets (i.e. {Z c } nc c=1 ) by minimizing the following function where Y c j denotes the score map of the j-th sample in the cth camera, and * is the convolution operation. The weights α c j ≥ 0 represent the importance of the j-th training sample of camera c, which is positively correlated with q c during tracking. In this formulation, training samples from all camera views contribute to filter updating, and thus enhance the robustness of the learned filters against view change.

Trajectory Prediction Network
Our key novelty in multiview tracking is the proposed Trajectory Prediction Network (TPN) for handling tracking failure using cross-view trajectory prediction. Intuitively, when the target is occluded (or damaged similarly) in a target view b, we can usually still reliably track the target in a different view, say, a source view a. Then, based on the geometric relation between the two views, we shall be able to locate the occluded object in view b from the trajectory in view a. The job, despite being nontrivial due to nonlinearity and camera movement, is done by TPN. Network design. Denote the trajectories at time t for view a and view b by g a t and g b t , respectively. It is natural for us to find the direct mapping and prediction between them. This idea does not work in practice due to the large range of absolute coordinate of object locations. Instead, we decompose a trajectory as a sequence of between-frame movements, denoted by r c t = g c t − g c t−1 as the the motion vector for camera c at time t. Then, TPN aims to map from r a t to r b t at time t.
At time t, based on 3D geometrical constrains, the object position g b t in view b can be transformed from its location g a t in a. Let d a t (or d b t ) and T a t (or T b t ) denote respectively the depth and transformation matrix for view a (or b). We can have the following derivation t d a t (T a t ) −1 and λ is for normalization, and t 0 is the begin time of a trajectory. Such relation between r b t and r a t motivates us to design the following Recurrent Neural Network (RNN)-based TPN model and introduce corresponding hidden parameters in the model: In the model, Θ enc (·) is an encoder network to translate/convert the input; Θ rnn (·, ·) indicates stacked RNNs to simulate the non-linear transformation decided by Q t and accumulate temporal information (e.g. t i r b i , g a t and object movement); p t denotes hidden states of RNN at time t and it initially encodes camera matrices T c t and the initial position g a t0 ; h p encodes the initial position g b t0 ; and Θ pos (·) decodes all features to output the results.  Figure 4. Prediction by TPN. The trajectory in the source view a is decomposed into direction and velocity and arranged in a motion vector. Then the encoder Θenc maps the motion vector into a 128dimensional representation, which then passes through Θrnn containing with 2 stacked RNN layers and 2 learnable hidden states hr 1 , hr 2 . Following that, another hidden vector hp is concatenated with output of RNNs and sent to PoseNet Θpos that captures the between-view geometric constrain. During prediction, only hr 1 , hr 2 and hp need to be updated. The trajectory in the occluded target view b is corrected by integrating the predicted motion vector R b t 0 ,t . Here, we only illustrate a data flow at a specified time t during prediction.
The structure of TPN is shown in Figure 4. The initial hidden state p k t0 of k-th RNN layer consists of zero vector and a learnable hidden parameter vector h r k , i.e., p k t0 = [h r k , 0]. These initial hidden states form p t0 = {p 1 t0 , p 2 t0 } together, where the hidden states p t of RNN in general encode temporal information. Moreover, we use h rnn = [h r1 , h r2 ] to represent all hidden parameters of RNN. PoseNet is a deep fully-connected network whose input includes both the output of RNN and a hidden parameter vector h p . Therefore, TPN can be viewed as a decoder that parses hidden parameters to a mapping function that maps motion vectors from source view into the target view. Note that Q t is treated as a dynamic transformation, and thus allows camera movement.
In practice, estimated trajectories from a source view often contain noise that may cause unstable trajectory prediction for the target view. For this reason, we smooth the source trajectories before sending them to TPN. Specifically, the smoothed motion vector r c t (we abuse the notation r c t for conciseness) is estimated by In this way, r c t consists of two parts, the velocity (left) and the direction (right). Trajectory Dataset. To train and test TPN, we first prepare a trajectory dataset by collecting trajectory pairs from different kinds of camera settings and object motions. In total, 25 scenarios are used for training and 8 for testing. The data of each scenario is captured by two cameras with different relative pose constrains. The between-camera relative pose may change slightly during capturing. An object is placed in front of the cameras. We move the object or the cameras randomly in the free space so that trajectory pairs are formed in different views without occlusion. We have 30,000 frames altogether and each sequence has at least 900 frames. Training TPN. As for training, we sample n b (n b = 100) trajectory pairs from the 25 scenarios per batch. The i-th trajectory pair (G a,i t0,t2 , G b,i t0,t2 ) is chosen from 90 continuous frames, i.e., t 2 − t 0 = 89 with t 0 randomly chosen. Using Eq.5, we get a motion vector set pair (R a,i t0,t2 , be the learn-able hidden parameters of i-th sample pair in networks and θ be other parameters of networks. Our objective is to find an optimal θ * that: where n t is the number of training pairs; Ψ(·, ·, ·) denotes TPN, which takes a set of motion vector R a , network parameters θ and learn-able hidden parameters h as inputs, and outputs R b for the target view. Θ int (·, ·) integrates motion vectors into 2D absolute positions according to given initial point. We can recover a predicted trajectory by Since h i is also unknown in the beginning of each batch, we divide the training process into two stages: Stage 1: We randomly initialize h i and conduct network training which only optimizes h i and fixes current θ for each training batch: where (R a,i t0,t1 ,R b,i t0,t1 ) is first 40 frames of (R a,i t0,t2 ,R b,i t0,t2 ), which means t 1 − t 0 = 39. Stage 2: We use h i * as initial parameters, and train networks parameter θ by using training samples in a batch.
We use the Rprop algorithm to optimize the network parameters. After 20 epochs of training on the trajectory dataset, we obtain θ * and finish training TPN. TPN in Generic Multiview Tracking. During generic multiview tracking, the situation may be complicated. There may be more than one unreliable and reliable views. At time t, for each unreliable view b (i.e. q b < τ ), we use TPN to estimate its trajectory. We also take the result of correlation filter, g b t , into account. The corrected object's location g b t for camera b is given by: where w = c,q c ≥τ q c is a normalized coefficient and Θ TP (·, ·) is a trajectory prediction function, which is TPN embedded and predicts the object location g b t in camera b. The behavior of Θ TP (·, ·) is defined in the following. For the input (G c t0,t ,G b t0,t1 ), G c t0,t1 and G b t0,t1 are used to train hidden parameters h * according to Eq. 8; After that, we can obtain G b t0,t = Θ int (Ψ(R c t0,t , θ, h * ), g b t0 ). Finally, we take g b t as the output result of Θ TP (·, ·). t 1 is the last time when view b is reliable. We choose 40 frames to train h * , which means t 0 = t 1 − 39. This equation builds connections among multiple cameras and guides the trajectory correction when occlusion occurs.
In reality, there can be no reliable views. In that case, we keep the last momentum of the target object in each view. Let r c t1 be the motion vector at the last reliable time t 1 in view c. g c t = g c t−1 + r c t1 .

Experiments
To evaluate methods in this task, we build a multiview tracking dataset and compare our tracker with others on it. Afterwards, we analyze the performance of our Trajectory Prediction Network. The experiment shows that the proposed networks can find out the relationship between two trajectories effectively and improve tracking performance. We also evaluate our approaches on PETS2009 [13].

Multiview Tracking Datasets
Due to the difficulty in collecting and annotating multiview scenes, there is a serious lack of multiview tracking datasets. The PETS2009 dataset [13], which contains sequences taking from eight cameras, is such a dataset. PETS2009, however, by itself is insufficient for convincing experimental evaluation with its low frame rate and resolution. For this purpose, we capture and manually annotate the Generic Multiview Tracking Dataset (GMTD) to facilitate relevant research and evaluation.
GMTD contains a total of 17,571 frames and consists of 10 multiview sequences with each of them captured by two or three synchronized uncalibrated cameras, under 1080p resolution and 30fps. During data capturing, cameras are either tripod mounted or hand held. In particular, for handheld cameras scenario, cameras may undergo small translation and rotation. Cameras are placed with different relative angles, for example, facing opposite or same directions, to form diverse trajectories in each view.
GMTD takes into account the diversity of scenarios and targets. Several different targets, including rigid ones (e.g. cans, lantern and basketball) and deformable ones (e.g. leaves, human and cat), were captured in the indoor, outdoor, artificial or natural scenes. During the acquisition process, a target may move under multiple camera views with more than 75% overlap. These 10 sequences mainly cover six aspects of challenges in visual tracking, including scale variation, motion blur, deformation, background clusters, fast motion and occlusion.
The selected target is manually annotated in each sequence by axis-aligned rectangle bounding boxes. The annotation guarantees that a target occupies more than 60% area of the bounding box. For further analysis, we also label target state in each frame as fully-visible, partially-occluded (33.73% per sequence on average) or fully-occluded (4.78% per sequence on average). Object is occluded by 20% to 80% in partially-occluded scenario. Others are fully-visible (if below 20%) or fully-occluded (if beyond 80%). Bounding boxes are predicted by human when the object is occluded. Some example images are in Figure 1.

Evaluation Methodology
We evaluate our method in two ways, with (see [21]) and without re-initialization respectively. With Re-initialization. Based on widely-used tracking performance measurements, we choose two easily interpretable measurements to evaluate methods, which are accuracy and robustness. When evaluating with reinitialization, 'Accuracy' refers to the area, in percentage, of the results overlaps with the ground truth and 'Robustness' is a probability of tracker failing after S frames. For traditional single view single object trackers, we apply syn-chronous tracking evaluations on each camera view of each scene individually .
For more details, the target bounding boxes in all views will be reset to the next nearest fully-visible frame once IOU drops to zero (the tracking result has no overlapping with ground truth bounding box) in any view. The tracker will be initialized by using new bounding boxes and frame images at the same time. Let a c i,t denote the IoU in view c at time t in scene i, V i = {t|∀c, a c i,t > 0} the valid set. The per-scene accuracy ρ i for scene i is defined as We run a tracker 5 times for each scene to obtain average accuracy ρ i . Thus, the overall average accuracy ρ is obtained by the weighted average as We visualize results in accuracy-robustness(AR) plots. In AR plots, each tracker is represented as a point in terms of its overall averaged accuracy and robustness on GMTD dataset. Comparatively speaking, The tracker performs better if it is located in the top-right part of the plot and worse if it occupies the bottom-left part.
Without Re-initialization. To simulate a more realistic tracking environment, we also test relevant trackers without re-initialization. Under these circumstances, the accuracy is defined as the same as re-initialization case. We denote δ i = |Vi| li as the success rate of scene i , where l i counts the total number of frames of the view in scene i . Similarly, tracker is visualized in a plot with respect to average accuracy and average success rate of each scene.
We compare our method with ECO tracker [8], which is a typical example of correlation based tracker. In our experiments, all trackers' parameters are fixed and τ = 0.5.

Results on the GMTD Dataset
With Re-initialization. The overall averaged accuracy of GMT is 0.6984 and the overall average robustness is 0.7477. On ECO tracker, the overall averaged accuracy is 0.7541 and the overall average robustness is 0.2985. We can see a huge improvement on robustness in most scenes. Results can be seen in Figure 6(left). Without Re-initialization. Tracking algorithms cannot apply re-initialization due to the absence of ground truth in the real world. Thus, we also conduct evaluation without re-initialization to evaluate the long-term performance of algorithms. The results are shown in Figure 6(right).

Trajectory Prediction Evaluation
We evaluate our proposed TPN on trajectories' test dataset and compare TPN with naive methods. Moreover, we also test variants of TPN to show that the current structure (TPN-S) is comparatively optimal.
We provide two naive methods for trajectory prediction. One copies speeds of reference view and integrates them into the trajectory of current view (Naive-C). The other one simply repeats the last average speed of current view's trajectory (Naive-S). Variants of TPN contain the standard TPN (TPN-S) described in section 3.5. Another variant's hidden parameters are removed and all network parameters are trained online during tracking (TPN-O).
We simulate trajectory predictions on test dataset in online tracking between two views, a and b, to evaluate these methods. During each simulation, we sample a trajectory pair (G a t0,t2 ,G b t0,t2 ) of two views from time t 0 to time t 2 (t 2 − t 0 = 89). The trajectory pair is divided into two parts. The first part is (G a t0,t1 ,G b t0,t1 ), where t 1 − t 0 = 39. We train these models by using this pair. Then we use the left trajectory G a t1+1,t2 of reference view a and trained model to predict the trajectory of the other view b, denoted G b * t1+1,t2 . After that, we use ground truth G b t1+1,t2 to calculate the pixel distances of each frame between G b t1+1,t2 and G b * t1+1,t2 . In order to prevent potential variance of performance in evaluation, we repeat the simulation 1000 times and obtain the statistic average error of pixels for predicted position in each frame.
Results are shown in Figure 5. This evaluation shows that our proposed TPN-S has the best performance in predicting trajectory from the reference view. Respectively, TPN-O, whose hidden layers are removed, suffers from over-fitting problem. Moreover, TPN-O updates all parameters online, which leads to more computational costs. There still are some failure cases for all methods, such as (d) in Figure 5. It failed because G b t0,t1 lacks sufficient information to infer the relationship between cameras.

Results on PETS2009
We also make a comparison on PETS2009 dataset. We trim five video clips from city center scene, which has sets of training sequences with different views. Video clips cover both sparse and dense crowd scenarios. We pick one pedestrian under two views of each video clip as targets and manually annotate ground truth bounding boxes for evaluation with re-initialization. Results are shown in Table 1.

Conclusion
In this paper we propose a novel generic multiview tracker (GMT) for visual object tracking with multi-camera inputs. Unlike most previous multiview tracking systems, our GMT requests no prior knowledge about the tracking target, allows camera movement, and is calibration free. Our GMT has two novel components, a cross-view trajectory prediction network and collaborative correlation fil-ter, which are effectively integrated with a correlation filter tracking framework. For evaluation, we contribute a generic multiview tracking dataset to alleviate the lack of proper multiview tracking benchmarks. In our carefully designed experiments, the proposed tracking algorithm demonstrates advantages over state-of-the-art tracking algorithms.