BRDF Estimation of Complex Materials with Nested Learning

The estimation of the optical properties of a material from RGB-images is an important but extremely ill-posed problem in Computer Graphics. While recent works have successfully approached this problem even from just a single photograph, significant simplifications of the material model are assumed, limiting the usability of such methods. The detection of complex material properties such as anisotropy or Fresnel effect remains an unsolved challenge. We propose a novel method that predicts the model parameters of an artist-friendly, physically-based BRDF, from only two low-resolution shots of the material. Thanks to a novel combination of deep neural networks in a nested architecture, we are able to handle the ambiguities given by the non-orthogonality and non-convexity of the parameter space. To train the network, we generate a novel dataset of physically-based synthetic images. We prove that our model can recover new properties like anisotropy, index of refraction and a second reflectance color, for materials that have tinted specular reflections or whose albedo changes at glancing angles.

Introduction
One of the main problems in Computer Graphics is the creation of synthetic images which are indistinguishable from real world objects, a process known as rendering. Among the existing rendering techniques, physically-based rendering algorithms achieve plausible results by computing the way light bounces through the scene, an operation in which the right definition of a material is critical to convey realistic results. To describe and encapsulate the material reflectance behavior, a common model is the a Bidirectional Reflectance Distribution Function (BRDF), but the acquisition and measurement of the BRDF of real materials is a big challenge due to the sheer number of uncontrolled optical phenomena affecting the light transport on the material surface. While there are specific techniques for this purpose [13,14,29,12,33], it is a slow, constrained and expensive process. A usual approach to circumvent this problem is to use a simplified analytic model controlled by a set of parameters [22,35,11,9]. Despite the expressiveness of these models, when trying to reproduce a real world material, this process of tuning parameters by hand is a time consuming and error-prone task for digital artists.
Anisotropic models are capable of depicting a great variety of materials ( Figure 2), however, choosing the appropriate model for the appearance is critical, and, sometimes, the flexibility required by an artist to tweak the parameters is not aligned with the facility of finding them in an optimization procedure. The approach of Aittala et al. [4] fits a learned model from Brady et al. [9] by means of a flash-no flash image pair and using non-linear optimization methods. In opposition to an empirical BRDF, in this work we chose a physically-based, industry-oriented model [1] widely adopted in the film and video game industries. Designed with artistic expressiveness as main goal, it has the key challenge of having a non-orthogonal, non-convex parameter space, making existing optimization methods unsuitable.
In this work we propose a novel real-time method based on neural networks to automatically estimate the BRDF parameters of an arbitrary material using just two low resolution images. We estimate ten parameters that model com- Figure 2: Examples of anisotropic materials plex reflectance effects, including anisotropy. As this is a highly ill-posed problem, we make some assumptions: we assume that pictures are taken in a controlled (but low cost) environment and, we also assume planar objects with homogeneous materials.
Recent methods based on deep learning techniques have proved to be successful in fitting the parameters of isotropic material models from arbitrary single images [28]. However, real world materials like brushed metals or cloth cannot be modelled with simple isotropic models like Phong [21] or Ward [34].
This paper makes the following contributions: • A method that estimates the parameters of an anisotropic BRDF from two shots of the material at 30 • and 90 • degrees.
• A hybrid deep learning architecture that combines feature learning with nested fully-connected layers that leverage dependencies in the parameter space of the model.
• A novel loss function that combines pure classification with gaussian smoothing to perform regression of the parameters.
• A novel synthetic dataset of planar samples of anisotropic materials rendered under multiple viewpoints.

Related Work
Anisotropic Materials The final appearance of anisotropic materials is determined by multiple factors including the mechanical properties of the material along with the viewer and light source orientation. Therefore, accurately capturing these complex interactions require the use of expensive gonioreflectometers [13,14], or controlled setups with various sampling strategies [29,12,33].
Similarly, a few parametric models exist to simulate anisotropic behaviors based on microfacets [35,11], Wasserstein coordinates [8], or non-physically-based models with editable capabilities [22]. Among them, the closest to our work is the method of Aittala et al. [4]. Equally relying on an image pair, they use Levenberg-Marquardt to optimize the orthogonal parameters of a machine-learned microfacets model [9]. In contrast, our anisotropic model is artist-friendly, meaning in practice that different set of parameters might yield the same appearance, and thus, making the problem unsuitable for conventional optimization methods. In this work, we handle this ambiguity by modeling the parameter relationships with a nested neural network that estimates the parameters of our model, given only two pictures of the material taken under two different orientations.
Other methods that model and capture anisotropy cannot be used in a wild and uncontrolled environment. We refer to the state-of-the-art paper of Guarnera et al. [16] for a complete analysis of anisotropic models and acquisition methods. We present the first low cost and easy to use a method that capture anisotropic materials using a highly expressive and artist-friendly model.

Isotropic Materials
On the other hand, isotropic materials, have received much more attention due to their simplicity to capture [27], model and render in virtual environments [37]. Non deep-learning methods for isotropic models include: capturing SVBRDF and normal maps from two shots [4], single shots [3], dictionary-based [18], or based on neural texture synthesis [2]. Others take as input RGBD images like [36] or [26].
However, recent works that rely on deep learning techniques have proved to outperform conventional optimization. We can categorized these methods in two big groups according to whether the loss function is computed as: 1) the pixel-wise difference between the ground truth image and a rendered image computed with the parameters predicted by the model, i.e. the render loss (rloss) [19,23,38,15,28]; 2) the difference between the ground truth and the estimated parameters, i.e. the parameter loss (ploss) [20,24,28]. Methods that rely on the rloss are in general more precise as the training learns how much a variation of parameters affect the desired results. However, they need the render function to be computed almost in real time. Our anisotropic model is computationally complex and takes several minutes to compute, making our network unfeasible to be trained with the rloss. We get inspiration from the work of Kim et al. [20], that takes as input an RGBD image, and uses the ploss enhanced with color histogram-based regularizators to find the five parameters of an isotropic Ward model [34]. On the contrary, we use a nested architecture designed to reduce the ambiguity given by the non-orthogonality of the parameters. With the exception of Meka et al. [28], the other methods estimate the material parameters by regression. Instead, we pose the problem as a classification task which results in more robust estimations with greater explicability. Table 1 Table 1: Comparison of deep learning-based methods for isotropic materials. All the methods take as input a single image except the method of Kim et al. [20] that needs RGBD images, the method of Li et al. [23] which constrains the input to planar samples of metal, wood or plastic, and our method, that requires two captures. Note that our method is the first to fit a complex anisotropic model, while all the other methods use isotropic material models.
Datasets for learning-based methods. Any supervised machine learning-based method requires an annotated dataset for training. Given the variety of models, each method has developed its own dataset specific for the task at hand without publishing it, with a few exceptions. The dataset of OpenSurfaces [6] with perceptual Ward [31] parameters obtained via crowdsourcing, SynBRDF [20] containing 500K RGBD images rendered under a variety of natural illumination (5000 materials, 5000 shapes), Objects under Natural Illumination Database [25] with six objects taken under five natural illumination environments with calibrated ground-truth geometry and illumination. These datasets rely on simple isotropic models of materials. Our MaxBRDF dataset, which we plan to make available, provides a very complex set of planar samples of anisotropic materials.

BRDF Model Analysis
We focused our work in the standard BSDF of the commercial software Maxwell Render [1], the reason is twofold: first, Maxwell Render is an unbiased physically based rendering engine, capable of generating realistic synthetic images reproducing a real setup [38]. And second, Maxwell's underlying BSDF is designed for artists, with very expressive but non-convex and non-orthogonal parameters: this is a challenging scenario for classic optimization approaches, but a nice testbed to explore the potential of CNN architectures for parameter fitting.
Maxwell Render's underlying BSDF is a reflectance (BRDF) and transmittance (BTDF) model defined by twenty-five (25) parameters. Although artists usually combine this model with spatially varying (SV) textures, often layered in a stack of multiple BSDFs, for the purpose of this work we match the material in isolation, although it could be extended to SVBRDF with further statistical analysis, such as the texture transfer technique described by Aittala et al. [4]. We focus our study in the reflectance component (BRDF) of the BSDF, as shown in Figure 3 and defined by the equations described next.
The BRDF defining the opaque reflectance is an adaptation of Ashikhmin-Shirley's model [5] able to represent complex behaviors like anisotropy and Fresnel reflections. The main difference is the elimination of the original diffuse term ρ d , and the inclusion of a 90 • -reflectance value which is dominant at glancing angles and the reversion to a purely diffuse model when the roughness is approaching 100%, as shown in Equation 1 which defines the BRDF model: Where f r is the Fresnel component, and ρ s is the specular component (See equations 3 and 4). Please note that the operations with RGB 0 , RGB 90 are not representing a simple multiplication of RGB channels, as the luminance is controlled by the Fresnel component. Only the chromatic hue is conserved, yielding an interesting feature for artists to represent tinted Fresnel reflections in materials such as metals.
Based on these equations, a wide range of materials can be represented by artists tuning the following ten parameters: • θ 0 : Roughness (R) determines the tiny imperfections that make a material reflect the light in a diffuse way. Its influence on the final appearance is very significant and most of the remaining parameters depend directly on its value. We can observe it for RGB 0 and RGB 90 in Equation 1 and 2, for anisotropy in equation 5 and for IOR (N d ) in Equation 2, as the term ρ s is multiplying the Fresnel term (f r (ω i )).
• θ 1−2 : Anisotropy and Anisotropy Angle address to how directional the light is reflected. Anisotropy is a common property in brushed metals. The n u and n v exponents depend on the value of R and anisotropy, and control the shape of the specular lobe in a similar fashion to the Ashikhmin-Shirley's model [5] as defined in Equation 5. The anisotropy angle controls the direction of the effect, by setting the origin w.r.t. the Φ angle within the plane defined by the surface U,V vectors.
• θ 3 : N d is the Index Of Refraction (IOR). It determines how light propagates through the material and the amount of reflection. A material with N d set to a low value will have weak reflections, while high values will create a mirror-like effect. The complex part of the IOR (the refraction extinction coefficient k) is set to zero in our data, due to its negligible effect in opaque materials.
• θ 4−6 : Reflectance 0 • (RGB 0 ) refers to the color of a material, i.e. how light is reflected when it hits the surface and is seen from a front view. It is composed of 3 parameters, the reflectance color in RGB space.
• θ 7−9 : Reflectance 90 • (RGB 90 ), like the previous one, its three parameters determine the amount of light that is reflected, but in this case, when it is seen at 90 • . The influence of each of the reflectance colors depends strongly on the roughness value R, but also on the N d value. In general, the RGB 0 value is more prevalent than RGB 90 on the final appearance, due to the form factor (cos θ o ) which adds weight to the camera-facing surface pixels and the use of a Lambert model for high values of R.
The equations show no interdependencies between Anisotropy and IOR(Nd), but in our analysis of the material appearance variation produced by fixing one parameter and varying the other for regular increments of R, the anisotropy-dependent term ρ s (ω i , ω o ) produced a greater variation per pixel for a given camera response. Intuitively, for increasing values of N d the existing reflections on the surface vary proportionally in luminance and color, while an equivalent differential variation on anisotropy, affects the shape of the reflection lobe ρ s , drastically altering the shape and position of the reflections over the surface (see supplementary material).
Optimization Strategy Any attempt to fit this set of ten non-convex parameters will face the problem of local minima produced by first, multiple possible combinations producing the same spatially local values, and second by overlaid appearance effects masking each other. In an ideal case we would try to detect and isolate the effect of one parameter over the others at a given pixel, but with traditional optimization approaches, this has proven to be challenging.
On the early stage of the research, we tried to fit the three parameters of RGB 0 albedo using a classical optimization method, but the results were limited. These preliminary results and the nature of our parameter space, led us towards deep neural networks, more suitable for such scenarios, materializing the architecture described in the next section.

BRDF Estimation
Our goal is to provide a low-cost tool to easily capture the appearance of complex anisotropic materials. To this

Feature Extraction
Nested Learning . . . end, we train a combination of deep neural networks that, taking as input two shots of the target material, yields the values of the ten parameters of our anisotropic model described in Section 3. Figure 4 shows the pipeline of our method. First, an initial pre-processing step transforms the input pair into a more suitable representation by means of computing its planar homography and whitening the images. The homographied images are fed into two CNNs that extract a set of features, which are the input to our nested neural network that predicts the material parameters.

Pre-processing
Our input consists of two views of the material captured at 30 • and 90 • , as we show in Section 5.1, we empirically observed that these two views alone summarize most of the interesting features of the material. Before feeding the input image pair to the network, we perform a pre-processing step. We leverage the constraint to planar materials and find the homography that warps each input material to the orthogonal camera plane, resulting in a squared image which we referred to as homographied input. This step facilitates the training and inference of the network particularly in real images that may have slightly different backgrounds. This transformation is shown in figures 5 and 10 for synthetic and real data, respectively. Additionally, the images are whitened to remove color correlations which hinder the estimation of parameters such as roughness and the index of refraction N d .

Parameter Loss
Before defining each of the modules of our pipeline, we describe the loss function that our network uses. Our goal is to predict the values for a set of parameters, a problem typically solved as a regression task [20,23]. However, by posing the problem as a classification task [28], we reduce the complexity: from having to find a number within the whole domain of the real numbers, to a fixed set of unknown values. We empirically found that discretizing each parameter into 100 uniform bins provides enough accuracy and convergence of the models. In contrast to most of the previous methods, which encode the material albedo using RGB space [28], we leverage the perceptually uniform distribution of CIELAB space and encode the RGB 0 and RGB 90 parameters in LAB. This enforces that a misclassified bin in this color space directly match human perception, improving the accuracy of the predictions, as we show in Section 5.
A vanilla classification loss function computes the crossentropy loss per each parameter θ i as where N = 100 is the number of bins/classes, x j is the probability for the class, and y j is the true label codified using one-hot encoding. However, in our particular scenario of regression-by-classification, a weakness of the loss defined in Equation 6 is that almost correctly classified parameters are dismissed. In other words, vanilla classification loss makes sense when there is no relationship between the classes, however, in our case, small variations in the parameters produce imperceptible variations in the render image. Therefore, mispredicted classes located close by in the parameter space should be rewarded. To this end, we apply a Gaussian 1-dimensional discrete filter of radius nine to the one-hot encoded labels Y (see Figure 6) and redefine our loss as:L whereŷ j is the value after application of the Gaussian smoothing to the whole vector Y . This change led to smooth results and patently improved the accuracy of our model.  Figure 7: Architecture of each of the two CNN of our Feature Extraction module. We obtain the feature vectors F 0 and F w corresponding to the output of the second convolutional layer for a network trained with the original data CNN 0 and a network trained with the whitened data CNN w

Feature Extraction
The first module of our pipeline aims at extracting highly discriminative features from the input pair. It consists of two identical convolutional layers with 64 and 128 5x5 filters, respectively. Each layer is followed by a ReLU (Rectified Linear Unit) activation, a batch normalization, an LRN layer (Local Response Normalization) and a max-pooling layer with a stride of 2. After that, we apply three dense (or fully connected) layers, all of them followed, again, by a ReLU layer. Between all the layers, we apply dropout (with 75% of keeping values) to make the network more robust and avoid overfitting. A diagram of the network is shown in Figure 7. Note that, in this step, all the ten classifiers are part of the same network that learns shared weights in the hidden layers and diverges in the last fully connected layer. Such structure is usually known as multitask learning and it improves the accuracy, as the learning of similar but different tasks can help the network to recognize relations and adds some noise that helps generalization [10].
The total loss to train this network is given by the sum of the local losses (Equation 7) per parameter as follows: where P is the number of parameters. We train two identical end-to-end networks: the CNN 0 network, which takes the original homographied pair as input; and the CNN w network, which inputs are additionally whitened. The performance of these two networks is reasonable, as our ablative study shows in Section 5, but still far from the ground truth. The reason is that the dependencies between the parameters are implicitly learned by the network, increasing the difficulty of the learning process. In light of these results, and as we know the parameter dependencies from the model definition in Section 3, we manually designed a nested fully-connected architecture that replaces anisotropy Nd RBG 0º RGB 90ª Figure 8: Nested Architecture. Each • represents a concatenation of features. Each FC contains three fully connected layers with the same structure as the CNN baseline architecture for feature extraction. Note that the feature vector F w is used as input for FC1 and FC2, and F 0 feeds FC3, FC4 and FC5.
the FC layers of the current CNN and combines the knowledge learned from both models, CNN 0 and CNN w .

Nested Learning
Given the feature maps F w and F 0 obtained in the Feature Extraction module, we train a nested fully connected architecture hand-crafted specifically for the task of estimating the parameters of our reflectance model. Each fully connected block (FC) has the same structure as the fully connected block of the CNN of the Feature Extraction module, i.e. three layers with intermediate units of sizes 1024 and 512 with a softmax activation unit per parameter of size 100. Each FC layer is trained independently and differ in the input and the output as described below.
From the definition of our BRDF model, in Equation 1, we observe that there is a dependency between the parameters, i.e. some parameters overwrite others. We explicitly model this dependency in the way feature maps and outputs of the FC layers are concatenated. The connections are depicted in Figure 8, where the dots mean a concatenation operation between vectors that intersect. For example, the roughness parameter θ 0 takes as input the feature vector F w and outputs 100 units; these units are therefore concatenated with the feature vector F w to form the input of the layer FC2. FC2 will learn two parameters θ 1 and θ 2 , which are again concatenated with the first parameter θ 0 and the feature vector F 0 to form the input of the layer FC3.

Dataset and Training
MaxBRDF Dataset It is well known that training deep learning methods require large amount of annotated data, however, annotating real world materials with their corresponding parameters of such complex model is unfeasible and error-prone even for skilled artists. We therefore present a new physically-based rendered dataset, the MaxBRDF dataset, with much richer annotations than existing works, which we use to train our networks.
Samples of the dataset are shown in supplementary material. We use Maxwell Render to generate 126K planar samples of 9K different materials varying our ten target parameters described in Section 3. For each material, we render 14 different views. First, with the light source set at 45 • of elevation above the sample, four rotations of the material with camera at 45 • , plus five orientations of the camera (15 • , 30 • , 45 • , 60 • and 90 • ), and finally, five additional views with the same camera directions but the light source at 90 • . In the former, we add a color checker, as the way the material reflects this colors can help detect some important features. The scene is illuminated with a spherical emitter that simulates a bulb with power value of 1000W and efficacy of 17,6 lm/W. The overall setup is simple and designed to be reproducible with real objects and material samples.
Training We use the synthetic data to train our neural networks. In particular, only the views at 30 • and 90 • with the color checker are used for training. We found that these views are sufficiently representative of most materials and we empirically proved that they are optimal for this task. The 12 other views are used at test time to validate the generalization of the predicted material to novel views (see supplementary material for an example). We separate 2/3 of the data for the training, 1/6 for validation and 1/6 for test. For some of the models, We perform data augmentation by adding Gaussian noise with a mean of 0 and random standard deviation between 0 and 4. We evaluate the performance of this operation in the following section.
We first train the networks CNN 0 and CNN w using the parameter loss of Equation 7 with Mini-batch Stochastic Gradient Descent, learning rate of 0.003 and random initialization of the weights. Then, we use the features learned in this first step to train the nested module. For this aim, the feature vectors are kept fixed (the weights of the convolutionals networks aren't updated) and the weights of the new fully-connected layers in the FC networks are randomly initialized and trained with a learning rate of 0.03.

Quantitative Evaluation
We quantitatively evaluate our method using the test set of our MaxBRDF dataset, from which we know both the ground truth parameters as well as the rendered images of each sample. Figure 9 depicts the pixel-wise mean square error (MSE) of the rendered predictions and demonstrates that our approach ("nested LAB") outperforms the baseline and other intermediate solutions.
In particular, we show MSE perpixel error for: first, our CNN 0 and CNN w alone, described in Section 4.3. As expected, these two networks are good at extracting discriminative features of the input images, but struggle to provide accurate predictions because they have to implicitly learn parameter dependencies; we also experimented with assigning explicit weights (in Figure 9, "custom weights") to each of the parameters to the loss function L total defined in Equation 8, hoping that these weights would ease the learning process. However, results were unsatisfactory, probably due to the complex parameter dependency of our BRDF model; finally, we show results with our nested approach using different representations to compute the error of the reflectance parameters RGB 90 and RGB 0 (in Figure 9, "nested RGB", "nested noise" and "nested LAB"). We conclude that our nested approach using the LAB color space to encode RGB 90 and RGB 0 , which includes Gaussian noise augmentation in the training set, outperforms any other method.

Qualitative Evaluation
The images of materials predicted by our method using the test set of MaxBRDF dataset are really accurate. Several images on how it generalizes to new viewpoints can be found in the supplementary material. However, our final goal is to create a method that estimates the parameters of  Figure 11: Qualitative evaluation on real materials. Each 4x4 block shows a pair of homographied images of a real material (captured at 30 • and 90 • ), and the predicted material rendered in the same configuration. This results demonstrates that the proposed method is capable of capturing complex appearance behaviors such as strong highlights and anisotropies.
pictures of real materials.
Even though our new MaxBRDF dataset features a large variety of physically-based rendered materials, it does not contain any real image to use for testing.
We overcome this limitation by capturing a new test set featuring real-world materials in a low-cost setup, as shown in Figure 10. Capturing the ground truth properties of those materials is a challenging task, even with the help of a goniorreflectometer and BRDF fitting methods which are beyond the scope of this work. However, we can use these images to visually assess the quality of the predictions and evaluate the potential of our method for casual BRDF acquisition on the wild.
For each material, we take a photo with a handheld camera approximately at 30 • and 90 • in a controlled environment, where a unique light is located roughly at the same position as in the synthetic dataset. As shown in Figure 10, following our pre-processing step, each of these image pairs is homographied before being fed into our network. Figure 11 shows 15 pairs of homographied real images of complex materials, and the corresponding renders of the predicted BRDF. We demonstrate that our method obtains highly accurate results, even for materials featuring complex anisotropic and specular behaviors. Furthermore, in the supplementary video, we show highly realistic animations of the predicted BRDF of real materials, using a moving camera. This demonstrates that our method can be used to render novel views of complex materials captured with just two quick shots.

Conclusions
In this paper, we have presented the first method to estimate the complex appearance of anisotropic materials using a novel architecture based on nested neural networks. We have shown that given only two views of a material sample at 30 • and 90 • degrees, our model can extrapolate the estimations to the remaining fourteen views of our dataset. We have shown the performance of our method with quantitative and qualitative experiments on synthetic and real data as input, providing compelling results in both scenarios. We have also presented the first method to achieve nonconvex parameter fitting by using a hybrid approach between feature learning with convolutional neural networks and classification-based regression using a nested fully connected architecture. We believe we are the first to provide such kind of solution to the problem outperforming end-toend baseline architectures.


Priorities in tock-CSP

The $tock$-CSP encoding embeds a rich and flexible approach to modelling discrete timed behaviours in CSP where the event $tock$ is interpreted to mark the passage of time. The model checker FDR provides tailored support for $tock$-CSP, including a prioritisation operator that has typically been used to ensure maximal progress, where time only advances after internal activity has stabilised. Prioritisation may also be used on its own right as a modelling construct. Its operational semantics, however, is only congruent over the most discriminating semantic model of CSP: the finite-linear model. To enable sound and compositional reasoning in a $tock$-CSP setting, we calculate a denotational definition for prioritisation. For that we establish a Galois connection between a specialisation of the finite-linear model, with $tock$ and $\checkmark$, that signals termination, as special events, and $\checkmark$-$tock$-CSP, a model for $tock$-CSP that captures termination, deadlines, and is adequate for reasoning about timed refinement. Our results are mechanised using Isabelle/HOL.

Introduction
Process algebras like CSP [1] enable modelling of reactive systems via named events that correspond to atomic and instantaneous interactions of interest. To specify budgets and deadlines, and to reason about liveness and safety over time, an explicit notion of time is required. Roscoe introduced the tock -CSP encoding, where an event tock is used to mark the passage of discrete time. Extensive use of tock -CSP has been reported [2][3][4][5][6].
The CSP model-checker FDR [7] has operators tailored for tock -CSP. In addition, it implements a Pri ≤ (P ) operator that can be used to prioritise events according to a partial order ≤. The behaviour is that of P , but changed so that whenever events a and b are available, then if b is of strictly higher priority than a, that is, a < b, then a, and the behaviour following a, is pruned.
Process R offers an external choice (✷) between behaving as a prefixing (→) on a or b, followed by immediate Email addresses: pedro.ribeiro@york.ac.uk (Pedro Ribeiro), james.baxter@york.ac.uk (James Baxter), ana.cavalcanti@york.ac.uk (Ana Cavalcanti) termination (Skip). Prioritising R, with a < b yields the process b → Skip.
Prioritisation can be used in FDR to enforce maximal progress, that is, that time can only advance after internal activity has stabilised, by prioritising τ , the internal action, and , which signals termination, over tock . Pri ≤ also endows CSP with extra expressivity [8], and has been applied also in abstraction techniques [9], reducing refinement in different CSP semantics to traces refinement [10], or as a modelling construct on its own [5,6].
The operator Pri ≤ has an intuitive operational semantics.
However, for Pri ≤ to be congruent over denotational models of CSP, namely the finite-linear (F L [1]) and refusal testing (RT [11,12]) models, the partial order needs to be constrained [8]. Thus, FDR actually implements a constrained form of Pri ≤ , where, for example, and τ are maximal in the order. However, this is insufficient for Pri ≤ to be congruent over weaker models such as -tock [13] and the stable-failures (F [1]). Process S makes an internal choice (⊓) between offering events a, or b, and terminating, or waiting a time unit (Wait 1) and then behaving as S again, specified using sequential composition ( ; ). Process T also makes an internal choice, but the choice between a and b is external. Although it may seem that T is more deterministic than S , deadlocking for a time unit, before making an internal choice again that may lead to the refusal of a and b, is a possibility for T . The -tock model does not distinguish S and T , just like the F model does not However, prioritising S and T , with a < b, assuming and tock are maximal in the order and that tock is not prioritised over b, yields different results. T becomes equal to a process priT = b → Skip ⊓ (Wait 1 ; priT ), whereas the prioritisation of S bears no effect. The incongruence arises as a result of Pri ≤ being defined over F L, where it distributes over internal choice. To reason about priorities in other models we need different definitions for Pri ≤ .
In this paper we calculate a definition of Pri ≤ for tock -CSP via a stepwise Galois connection between the F L model and the -tock model [13], the only sound model for tock -CSP that can be used to reason about timed refinement, and that captures deadlines, termination and erroneous Zeno behaviour. It is a faithful account of the tock -CSP dialect and is mechanised in Isabelle/HOL [14]. Included in this work is a mechanisation of the F L model, that handles termination, via the special event , and a specialisation that includes tock . This paper is structured as follows. In Section 2.1 we describe -tock and F L. In Section 3 we formally define F L. In Section 4 we define a Galois connection between a specialisation of F L, which includes tock as event, and calculate the induced definition of Pri ≤ . We conclude in Section 5.

Models
Here we describe the -tock and F L models by summarizing material from [13] and [1].

-tock
The -tock model is defined in terms of a set Σ of events, not including and tock . The complete set is defined as Σ tock . The semantics of processes is a set of sequences of observations of type Obs, defined below. An observation is either an event in Σ tock , or the refusal of some subset of Σ tock , so Obs has two constructor functions evt and ref .
The type of valid traces is defined as TT , which is the set of all sequences t with elements of type Obs that satisfy three conjuncts, defined below.
The first conjunct ensures that can only appear as the last event of t . The second conjunct requires that every refusal before the last (i < #t ) is followed by a tock . Finally, the third conjunct ensures that every tock is preceded by a refusal.
The healthiness conditions of -tock [13], whose composition is named TT, ensure properties of the standard models of CSP in the context of TT traces. Namely, the empty trace is an observation of every process; prefix closure and subset closure of refusals; and an event that cannot be performed is refused. Finally, whenever there is a stable refusal in a trace, then there is always at least one (other) trace where the refusal includes .

Finite-linear
The F L model is characterised in [1] by finite sequences A 0 , e 0 , . . . , A i , e i , A i+1 of alternating acceptances A i and events e i . An acceptance is either a set, recording the events being offered, or null (•) indicating the impossibility to observe such a set because of instability. An event e i necessarily belongs to A i if A i is not •. Valid sequences end in an acceptance, or • followed by . The healthiness conditions of F L ensure prefix closure, and require that, whenever an acceptance A i is observed, then any events in A i can also be performed.
The prefix relation for sequences in this model allows • to precede a stable acceptance set, so that • and •, e 0 , • are prefixes of A 0 , e 0 , A 1 , for example. Crucially, and unlike other CSP models, there is no upward-closure of acceptance sets.

Formalising the F L model
In Section 3.1 we define a recursive data type to capture F L traces. In Section 3.2 we formalise the healthiness conditions. Finally, in Section 3.3 we formally define Pri ≤ . While Roscoe [1] characterises the F L model and studies in depth its relationship to other CSP models, here we define its data model and healthiness conditions with the level of formality required to mechanise it (as an Isabelle theory as presented in [15], for example).

Model
An acceptance is either a null acceptance or a set of events. It is defined by the type Acc, which has two constructor functions • and aset , respectively, where is the Z [16] syntax for constructors.
Definition 3. Acc ::= • | aset P Σ tock We also define: e ∈ F L A to be true exactly when A is the result of applying aset B for some set B and e ∈ B ; and a prefix order on Acc, where • is the least element, and aset A ≤ aset A, which can be lifted to define the prefix order on FL traces. An acceptance followed by an event is a pair of type Aev associating elements of Acc to Σ tock .

Definition 4 (Acceptance-event).
Aev : Acc × Σ tock ∀ B : P Σ tock ; (aset B , e) : Aev • e ∈ B An acceptance-event pair (aset B , e) is valid exactly when e is a member of set B . A pair (•, e) imposes no condition on e. Next we define the type of nonempty traces for F L as FL, a recursive data type with two constructors, acc and aev . The function acc takes a single acceptance while aev takes an acceptance-event pair and an FL trace. Processes in F L are defined by a set of FL traces, effectively finite non-empty sequences ending in an acceptance. Unlike the original presentation of F L, we consider as a regular event. This simplifies the type definition of FL and allows us to give a general account of Pri ≤ where can be prioritised.
To facilitate presentation we abbreviate acc A as A F L , and aev ((A, e), ρ) as (A, e) # ρ. Furthermore a recursive application of # a number of times, such as (A 0 , e 0 ) # (. . . # A i ) is abbreviated as (A 0 , e 0 ), . . . , A i F L . This is a typical approach to encoding finite lists via recursive data types.

Name
Definition

Healthiness conditions
The healthiness conditions are listed in Table 1. The first, FL0, although not listed in [1, p.257], is required to ensure that every process has some behaviour. Together with FL1 (adopted from [1]), which ensures prefix closure, FL0 ensures that every process has at least the trace • F L .
The next condition FL2 from [1] is restated using ∈ F L and a concatenation operator F L that is closed under FL, unlike the general sequence concatenation operator. FL2 states that whenever a trace ρ concatenated with an acceptance A is in P , then for every event e in A there must be a trace in P that performs e. It is the result of concatenating ρ with the trace consisting of the acceptance-event pair (A, e) followed by •.
To ensure is only possible after • as the very last event of a trace, FL3 requires valid traces to be in a set FL . It consists of FL traces where is not offered in any acceptance and, if appears, it is as the last event followed by •.
The F L model forms a complete lattice under the refinement order defined by subset inclusion. The top is the process div whose only observation is the trace • F L , while the bottom is FL , the set of all possible observations: the process Chaos(Σ).

Prioritise
The semantics of Pri ≤ (P ) is defined in [8] pointwise over the FL traces of P as follows.
It is the set of traces ρ related by pri ≤ to a trace σ drawn from P , that is, a trace ρ is a possible prioritisation of σ. The relation pri ≤ is a fully specified version of that presented in [8], generalised to cater for any event, including , and defined inductively in Definition 7 as required for mechanical reasoning.
Observe that pri p is parametric over an arbitrary partial order p on events. Before explaining the definition, we illustrate the calculation of traces ρ, as related by pri ≤ , from traces σ of process R (of Example 1), assuming a < b and that is maximal in the order. The set of traces of R is presented below, as well as those of Pri ≤ (R), where we omit the F L subscripts. We observe that the result is independent of whether b is maximal in the order.
We have traces: aset {a, b} recording that both events a and b are stably offered, and traces such as (aset {a, b}, a), (•, ), • recording that after stably offering both events, and performing event a, then termination, encoded by happens unstably, and similarly for event b; and finally traces that complete the prefix closure as required by FL1.
Prioritisation of R with a < b, as previously discussed in Section 1, leads to pruning the behaviours of the prefixing on a. The traces of Pri ≤ (R) are those related to the traces of R by pri ≤ .
The trace • is related to itself as it is a valid observation of every process in F L. The trace aset {b} is related to aset {a, b} , as {b} is the set obtained by eliminating events in {a, b} for which an event of strictly higher priority is also in the set, namely a is eliminated as b is of higher priority. Similarly, the trace (aset  pri In general, a trace A is related to Z by pri p , whenever A is priacc p (Z ), specified below, that defines the result of prioritising an acceptance Z . In Example 3 above, we have that priacc ≤ (aset {a, b}) = aset {b}.
Definition 8. priacc Formally, priacc p (•) is •, whereas for an acceptance set Z , priacc p (aset Z ) is the acceptance set of events in Z for which no event b of strictly higher priority, according to the order p, exists in Z . A trace (A, e) # ρ is related to (Z , e) # σ by pri p exactly when A is a prefix of the acceptance permitted by prioritisation, as defined by priacc p (Z ), ρ is related to σ by pri p , and if the event e is not maximal in the partial order p, where max (p, e) = ¬ ∃ x • e < p x , that is, no other event has strictly higher priority than e, then e must be in the acceptance permitted by prioritisation. In other words, events of maximal priority are never eliminated, whereas those for which an event of higher priority exists need to be in the resulting acceptance set obtained via the intersection. In particular, if Z is • and e is not maximal, then it is not related by pri p . Although our definition of pri p is rather concise, we have established using our mechanisation that it is equivalent to that of Roscoe [8].
A formal definition for pri ≤ suitable for mechanical reasoning enables key results about Pri ≤ to be established, namely that it is monotonic and closed under the healthiness conditions. Using our mechanisation we have established the following novel results where the subscript F L indicates that these are operators of the F L model.
The first result in Lemma 1 establishes that Pri ≤ is idempotent; the second that it distributes through internal choice; and the third that it distributes through prefixing. It also distributes through sequential composition, as established next.

Lemma 2.
Provided has maximal priority, In this case must be maximal in the order, as otherwise the possibility for P to terminate could be eliminated. More interestingly, the prioritisation of an external choice between prefixings on events a and b, where a < b, is the prioritisation of the prefixing on b, with any behaviour following on from a pruned, as established by the lemma.
Proof of this, and other key results, is available 1 .
Our formalisation of F L and Pri ≤ are used next to calculate a definition for Pri ≤ in -tock .

Galois connection
The key to defining a Galois connection between F L and -tock is relating FL traces, which record acceptances before every event, and TT traces, which instead record subset-closed refusals, and only before tock events or at the end of a trace, as previously discussed in Section 2.1. We define the Galois connection stepwise to simplify proofs. In Section 4.1 we first consider a Galois connection between F L and a variant of -tock where refusals sets are not subset closed; instead they are maximal: they record exactly what is refused. In the following Section 4.2 we define a Galois connection with full -tock by completing the subset closure of refusals. Fig. 1, which is explained as we discuss each step, provides a depiction of the connections. Finally, in Section 4.3, we present the result of calculating the induced definition of Pri ≤ for -tock .

From F L-tock to maximal -tock
The pair of functions fl 2ttm and ttm2fl , mapping between F L-tock and the maximal variant oftock , labelled as TTM in Fig. 1, is defined below.
The function fl 2ttm is defined as the set of traces obtained by applying fl 2ttobs, a total non-injective function from FL to TT traces, defined next, to every trace ρ in P . The inverse mapping ttm2fl is uniquely defined in terms of fl 2ttm, and is the distributed union over the set of FL processes that can be mapped via fl 2ttm to be a subset of P . This construction is standard for Galois connections where one adjoint uniquely defines the other. The function fl 2ttobs is defined as follows. It maps: the null trace • F L to the empty trace , since -tock does not record instability; and every trace whose only element is an acceptance set aset A to a singleton trace consisting of a refusal obtained as the set complement of A. The mapping of an acceptance-event pair followed by a trace ρ is split into three cases: (1) acceptances a preceding regular events e, other than tock , are mapped to a trace evt e whose only element is the event e, concatenated with the result of applying fl 2ttobs to ρ, since -tock does not record refusals before such events; (2) for the same reason, null acceptances preceding tock are mapped to the empty trace; (3) acceptance sets preceding tock are mapped to a trace consisting of a refusal set, obtained as the complement of A, followed by the event tock , concatenated with the application of fl 2ttobs to ρ.
The functional application of fl 2ttm to a healthy process P satisfies nearly all the healthiness conditions of -tock , as established by Lemma 4. It also satisfies extra healthiness conditions, defined below. They are relevant to obtain the final Galois connection in the next section. Namely, fl 2ttm(P ) satisfies TT0 and TT2-4, but not TT1 [13], which requires subset closure of refusals. This is expected, as fl 2ttobs is a function. (The subset closure of refusals is completed by the connection in Section 4.2.) Instead, we have prefixclosure of sequences, ensured by a healthiness condition we call TT1w, a weaker form of TT1 that does not enforce subset closure of refusals.
In addition, refusals are maximal, and so fl 2ttm(P ) satisfies the additional conditions TTM1-3 listed in Table 2. TTM1 requires that every event e that is not refused can be performed. TTM2 is similar, but requires tock to happen after the refusal. TTM3 requires that P is a subset of TT , the set of all traces where every refusal contains . The three conditions, together named TTM (see Fig. 1), characterise the set of -tock processes whose refusal sets contain exactly the events refused.
Since fl 2ttm is monotonic with respect to the refinement order, and ttm2fl is closed under the healthiness conditions of FL, we have a Galois connection between F L-tock and maximal -tock .
Next we focus on the subset closure of refusals to complete the Galois connection.

Subset-closure of refusals leading to full -tock
Completing the subset closure of refusal sets in -tock is achieved by the function mkTT 1.

Name
Definition Definition 11.
With mkTT 1(P ), we have the union of P with a set of traces ρ related by , the prefix relation on -tock traces defined in [13], to σ drawn from P . The other adjoint is unTT 1 and is uniquely defined in terms of mkTT 1. It is the distributed union over processes Q (where Q satisfies TTM and TT1w) and whose mapping via mkTT 1(Q ) is a subset of P . In other words, unTT 1 undoes the subset closure of refusals resulting from mkTT 1, yielding a process whose refusals are maximal, as defined by TTM, and whose traces are prefix-closed under TT1w.
Thus, we have a Galois connection between maximal -tock , labelled as TTM in Fig. 1, and full -tock . Next, we compose both connections to obtain a Galois connection between F L and -tock , depicted in Fig. 1 by the dashed arrows.
The function fl 2tt (P ) maps F L processes totock . In the opposite direction we have tt 2fl .

Prioritise for -tock
Similar to the definition of Pri ≤ , which is pointwise over FL traces using a relation pri, we have a similar characterisation for Pri TT≤ as defined next.
The traces of Pri TT≤ is the set of TT traces ρ related to a trace σ, from P , by a parametric relation pri TT[≤, ,P ] , specified in Definition 14, that takes three parameters: the partial order ≤ over events in Σ tock ; the empty sequence , and P .
For a trace σ, prioritisation needs to take into account, at each point in σ where an event is present, the refusal sets that can be observed just before it, as well as whether other events can be performed. So the second parameter of pri TT is the trace considered so far in the process specified as the third parameter. In the initial case, when considering σ, the trace considered so far is the empty sequence. Before explaining the formal definition, we illustrate the calculation of Pri TT≤ (T ) (from Example 2), where a < b, and tock have maximal priority, and ¬ b < tock . The trace is related to itself as it is a valid trace of every process (see 1 in Definition 14). The trace ref { } is related to itself, and every trace whose only refusal is a subset of { } is also related to it because of subset closure (see 2 ). Similarly, ref {a, b, } is related to itself, and to traces whose only refusal is a subset of {a, b, }. The trace evt a is equally related to itself (see 4 ), as although a is not maximal, the presence of the trace ref {a, b, } indicates that b could be refused, in which case a is possible on its own, and thus prioritisation has no effect. Similarly evt b is related to itself as b is prioritised over a, can only happen unstably, and tock is assumed not to be prioritised over b. The traces evt b, evt and evt a, evt are similarly related to themselves, given that has maximal priority. Finally, because tock is maximal, priref (p, σ, Q , S ), which is specified below. It defines a refusal set containing all the events of S as well as those disallowed by prioritisation, according to the order p, given that the trace preceding ref S in Q is σ. For example, the result of applying priref (p, , T , ∅) is {a}, assuming a < p b, as event b can be performed in T , but is of higher priority than a and b is not in the empty set. Initially, in the context of Pri TT , σ is the empty trace.
The function priref is defined as the union of S and two sets: the set of events e of strictly lower priority than tock , according to the order p, if the trace σ concatenated with the trace where the refusal S followed by tock is in Q ; and the set of events e such that there is some event b of strictly higher priority than e, which is not tock nor and is not in S , but that can be performed in Q after trace σ as σ evt b . Thus, to ascertain whether an event e can be refused because of prioritisation, it is necessary to determine whether an event of higher priority, other than , that is not refused in S can be performed after trace σ. The reason is excluded from the comparison is because it is an unstable event, and so it is never in a genuine offer to the environment. To illustrate this point, we consider the process in the following example.
In addition to e being offered stably, we have the possibility that process K terminates. Assuming f < p , applying priref (p, , K , ∅) would result in the set {f } if f could be compared with in priref . Because is never a genuine option for the environment, however, it is incorrect to compare it with events of lower priority. Moreover, as illustrated in this example, the set of traces induced by such a definition of priref (p, , K , ∅) would be unhealthy, as no refusal set f , could be obtained by applying priref to an initial refusal of K (see Section 2.1). The final case 4 in Definition 14 specifies that a trace evt e ρ 0 is related to evt e ρ 1 by pri TT[p,σ,Q] , whenever ρ 0 is related to ρ 1 by pri TT[p,σ evt e ,Q] and, if e is not maximal in the priority order p, then it is not and there is some refusal Z observable after σ in Q , such that application of priref (p, σ, Q , Z ) does not lead to e being refused. For example, trace evt a from process T is related to itself when considering a < p b, as although a is not maximal there is a trace ref {b} in T , by subset closure, where the application of priref (p, , T , {b}) yields {b}. In words, if e is maximal then the trace is related, and in particular, if appears in a trace it must be maximal. Otherwise, if e is neither maximal nor , then it must be the case that application of priref to Q indicates that e can be chosen; not refused.
Thus, events that are not maximal are not discarded by prioritisation if, at the same time, events of higher priority can be refused. We consider, an alternative presentation of T , by distributing the internal choice over the external choice.
This clearly shows the possibility for a to be stably offered for a time unit, if the first internal choice is decided in favour of a, while the second internal choice is made in favour of a delay. Thus, before the first time unit has elapsed, there is the possibility that no b is on offer, and thus prioritisation is ineffective at that point. Another presentation of T is exactly S , which, as discussed in Section 1, has the same semantics as T in -tock , but not in F L. This is inherently related to algebraic properties enjoyed by -tock processes, which are consistent with the F model within a time unit.
Our main result is that the Pri TT≤ operator of -tock is exactly that induced by the Galois connection with F L as established next. Theorem 1. Provided P is TT-healthy, Pri TT≤ (P ) = fl 2tt • Pri ≤ • tt 2fl (P ) For a -tock process P , Pri TT≤ (P ) can be obtained by mapping P into F L, via tt 2fl , then ap-plying the prioritisation operator of F L, and finally mapping the result into -tock via fl 2tt .

Conclusions
To endow -tock with a prioritise operator consistent with Pri ≤ of F L, we have established a Galois connection between F L and -tock , and calculated an induced definition. With that, we have a definition that is by construction monotonic and healthy. It is strictly not pointwise because prioritisation of events in a trace needs to take into account other refusal sets reachable at the same point.
The formalisation of F L, although presented here in terms of a set Σ tock , is fully parametric in our mechanisation in Isabelle/HOL, and thus depends only on Σ. The mechanisation of the F L model is layered. We have a faithful mechanisation of F L, built stepwise to consider termination, and discrete time, by including and tock incrementally. The algebraic laws from Section 3 for Pri ≤ are novel, and more importantly are applicable to RT as a consequence of Roscoe [1]'s results on the hierarchy of CSP semantic models. Congruence of the operational semantics of Pri ≤ over that model, however, requires further restrictions to the order [8], including that is maximal, enforced, for example, when model-checking with FDR.
Effectively, Pri TT≤ can look into nondeterministic choices unlike other CSP operators. This presents challanges for its implementation in FDR. It is known that Pri ≤ cannot be specified via combinators, as used in FDR, and instead an extended theory of combinators is required [8]. Thus, Pri ≤ is actually implemented in FDR as a function that operates over an LTS, rather than via combinator semantics. A practical strategy for implementing Pri TT≤ in FDR, for example, may require some form of bisimulation, or search, to examine refusals reachable by sequences of τ transitions from each state being prioritised in the LTS.
A denotational definition for Pri TT enables algebraic properties to be explored. It is in our plans to propose laws in support of a refinement approach to verification of robotic simulations [6], where prioritisation is used as part of capturing the assumptions routinely made by roboticists in simulations.
Finally, the semantic model -tock is somewhere between F and RT in terms of expressivity, as indicated by the way refusals are recorded. Our results are likely to be enlightening in the calculation of a counterpart of Pri ≤ for F as well.


Underwater Object Detection and Pose Estimation using Deep Learning ?

This paper presents an approach for making a dataset using a 3D CAD model for deep learning based underwater object detection and pose estimation. We also introduce a simple pose estimation network for underwater objects. In the experiment, we show that object detection and pose estimation networks trained via our synthetic dataset present a preliminary potential for deep learning based approaches in underwater. Lastly, we show that our synthetic image dataset provides meaningful performance for deep learning models in underwater environments.

INTRODUCTION
Recently, object detection and 6D pose estimation have utilized deep learning methods. These methods have shown significantly impressive results for general object recognition tasks in unconditional environments and have sufficient accuracy for robotic tasks, including grasping objects ; Chu and Vela (2018)). However, the results of previous research mostly focus on terrestrial environments. Since datasets constituting underwater environment are scarce, implementing a deep learning based approach for underwater applications is challenging.
Obtaining and utilizing underwater object data lead to two further issues. Firstly, acquiring underwater object data is challenging compared to acquisition from the ground environments. Even if the data are obtained, manual annotation is costly and could have inaccuracies due to human error. Secondly, underwater camera images should have diverse variations such as intensity degeneration and color distortion (Chen et al. (2017)).
In this paper, we enhance the existing approach, in which a dataset is created by adopting a 3D computer aided design (CAD) model, to ensure that the dataset involves various optical conditions and underwater environments. To meet the research objective, we present an automatic annotation tool. Also to verify the effectiveness of our dataset, we show its application to object detection and pose estimation. In addition, we propose a simple pose estimation network. The object detection network is trained with our dataset and presents preliminary potential for deep learning based approaches in underwater applications. We verify that our dataset suits deep learning model in underwater environments.
This study is a part of the results of R&D project, Development of Basic Technologies of 3D Object Reconstruction and Robot Manipulator Motion Compensation Control, supported by KRISO. Fig. 1. Illustration of synthetic image generation. The 3D model is projected by a virtual camera (Virtual Camera (V-CAM)) to capture synthetic images. By extracting the annotations for the object detection and pose estimation networks from these synthetic images, we utilized these synthetic images with object mask and object class annotations for the training set of the object detection network. Then, we cropped the synthetic images using truncation annotation. These cropped images and pose annotations were piped into the training set for the pose estimation network.
In summary, this paper presents three things as follows: • We automatically generate all the necessary annotations for object detection and pose estimation. • We propose a simple pose estimation network for underwater. • We verify that the synthetic image set using a 3D CAD model is feasible for training in underwater object detection and pose estimation.

RELATED WORKS
Securing enough training data is essential for deep learning based approaches. In fact, since acquiring diverse datasets is challenging, many researchers have focused on creating synthetic datasets to enable automatic annotation.

Synthesizing Images Using a 3D CAD Model
In the literature, authors have utilized synthetic images generated using 3D CAD models to detect objects and estimate poses. Peng et al. collected 3D CAD models by searching online for the names of 20 categories. In their paper, 25 models per category were coated with a texture, and the authors selected their colors. The viewpoint was changed manually to render virtual images during the model generation step. Su et al. (2015) selected 3D models from PASCAL 3D+ for 12 categories. They randomly adjusted the illumination condition, viewpoint, and background when generating the images. Our work is similar to Su et al. (2015), but we focus on underwater implementation.

Automatic Annotation Tool
Precise annotation of the target to be learned is essential in supervised manners to ensure the significant performance of deep learning based approaches. Manual annotation is the most common method of addressing this issue. However, this annotation scheme can be inaccurate, and infeasible for large datasets. Even though researchers have used automated annotation tools using a real image to overcome this issue, automated generation of instance annotations in images has been a challenging issue caused by the impediment of occlusion with other objects. A synthetic dataset can alleviate these problems.  (2018); Wang et al. (2018) developed automated annotation tools using 3D CAD models. Our approach automatically annotates viewpoints, bounding-boxes, and segmentation labels for use in underwater environments.

METHOD
We composed a network comprising two cascadely connected networks, (i ) Mask R-CNN and (ii ) a pose estimation network to perform instance level object detection and pose estimation. The first network, Mask R-CNN, detects mask, bounding-box, and class for an object. For input to the pose estimation network, an image is truncated by bounding-box of an object. This input truncated by bounding-box, thus, allows the pose estimator to focus only on the objects for which the pose is to be estimated. Secondly, the proposed pose estimation network was combined with DenseNet (Huang et al. (2017)), Dense Block, and FC.

Synthesizing the Image
The entire synthesizing stage was processed automatically without human intervention. As shown in Fig. 1, a 3D model was located in the center of the spherical coordinate system. The 3D model remained stationary, only the pose of the Virtual Camera (V-CAM) was changed to acquire samples for relative poses between the V-CAM and 3D model (i.e., azimuth, elevation, in-plane rotation, and distance). The V-CAM parameter was tuned manually. At the same time, we obtained a transparent background image with the model centered using the V-CAM for every pose sample. This transparent background image was utilized for extracting the truncation parameters and segmentation labels. For the training set of the object detection, we overlaid a background onto the generated transparent background image. For the training set of pose estimation, after cropping the transparent background images using the truncation parameter, we coated the cropped images with the background. these image were employed for the training set of the pose estimation. Finally, we augmented all of the images using various effect to be robust in unconditional environments.

Pose Estimation
In this paper, since pose estimation was used to validate our synthetic dataset, we did not estimate full 6D poses but only the 3D orientation. Thus, we wanted to determine whether our synthetic dataset is appropriate for the pose estimation task. The proposed pose estimation network belongs to the regression task. In the rotation regression task, we used quaternion rotation representations because they do not suffer from gimbal lock, which occurs frequently in Euler angle representation, and by construction, quaternions are unit-norm, q 2 = 1. The cost of quaternion regression is shown in (1).
q is the ground truth value, andq is the predicted value. The operator · , · indicates the inner product. The prerequisite of this cost is q,q < 0 to prevent quaternion ambiguity.
To approximate the four parameters in quaternion, we created a new network by combining one DenseNet, four Dense Blocks and four FCs. DenseNet (Huang et al. (2017)) extracts more complex key points from the learning process than other networks do because almost all of its layers deploy the information of the previous layer through a skip connection. All parameters have shared features through the DenseNet. We assign a Dense Block to each of the four parameters; each block extracted key points for one parameter. One FC was allocated for each parameter (Fig. 2).
In fact, object pose estimation from a single RGB image is known as a substantially challenging task. To solve this challenging task, we needed to build a more specific model and costs focused on orientation information like Kehl et al. (2017); Do et al. (2018); Xiang et al. (2017). Our purpose was effective evaluation of our synthetic dataset for pose estimation task, by judging if our synthetic dataset has the potential for pose estimation from simple network.

Experiment Setting
We made a training set utilizing four 3D CAD models. For the underwater experiments, we prepared 3D physical models using a 3D printer. Then, we fed these outputs into a water tank to validate the preliminary performance. (Fig. 3).
For training, we generated 1000 samples for each model for the object detection and 2000 samples for the pose estimation. The example results for the above process are shown in Fig. 1. When the object detector and pose estimator were trained, we exploited only synthetic images. We used uncropped images for the object detector and cropped images for the pose estimator as shown in Fig. 1.
For the test, we captured three images per object (i.e., DUCK, RABBIT, FISH, and CHESS), in differing poses, along with images containing all four objects (ALL), as shown in Fig. 4.

Object Detection
Mask R-CNN used in the experiment is a state of the art object detection model that provides the class, mask, and bounding-box for an object. We evaluated whether our synthetic dataset is suitable for underwater object detection using Mask R-CNN. We set the threshold of the detection confidence to 0.9. As the evaluation metric, we used Average Precision (AP) and the pixel-level overrode ratio of the ground truth mask and predicted mask to measure the object detection accuracy (Table. 1).
Except for fish, the AP and mask overlay of the objects exceeded 0.9. Also, when we analyzed the images as shown in Fig. 4, the detection scores (detection confidence) were close to 1.0, and the bounding-boxes and masks fitted fairly. The object detection results trained by the proposed method performed well for various illumination conditions. The object was reliably detected even in the over-exposure situation shown in Fig. 4 without requiring preprocesses such as deblurring or dehazing.

Pose Estimation
Both the objects and the camera were in a water tank during the experiment. To evaluate the pose estimation results, we made a test set using 1000 synthetic images and used the mean absolute error between the ground truth and the predicted value.
As shown in Table. 2, quaternion as the output of the pose estimation network were converted to Euler to improve their readability. As can be seen, the roll and yaw were somewhat larger than the pitch, by nearly an average of 20 • . Since the pose error and the deviation of the error between the objects were not large, we can conclude that our synthetic dataset has preliminary potential for pose estimation. On the other hand, the experimental results revealed a prevalent 180 • difference between the ground truth and the estimated value of one axis. Our pose estimator could be confused when distinguishing the front and back of an object, which is a common problem in pose estimation with regression. Thus, we need to build a more specific model focused on orientation information and costs that copes with object symmetry.

CONCLUSIONS
In this paper, we proposed an approach to making a synthetic image dataset with an automatic annotation tool using a 3D CAD model. We applied this dataset to object detection and pose estimation in underwater environment. The experiment proved that our synthetic dataset presents preliminary potential for underwater object detection and pose estimation. In future works, we will pursue improvements in the dataset for the pose estimation. Furthermore, through the pose estimation quality improvement, we seek to grasp the object in underwater environment.


extfieldsofstudy: Physics Engineering

text:
Cipularang toll road safety audit of traffic signs and road markings

The Cipularang Toll Road is built to shorten the distance and time travel for people from Jakarta to Bandung. The Toll Road has been used since 2005 and the number of vehicles going through it keeps growing every year. With the increased amount of vehicles passing through the Toll Road, the risk of accidents occured also rises. Special aspect of concern must be given to traffic signs and road markings on the Cipularang Toll Road. This research is, conducted by data through the observation of traffic signs and road markings, and then it is compared with the regulations of road signs and markings according the regulations of Ministry of the Transportation. The conclusion drawn is that the absence of traffic signs and road markings resulted in an increase of accidents.

Introduction
The increase in the number of motor vehicles due to the ease of owning a vehicle and has been caused of traffic accidents to increase both in terms of quantity and level of fatality. The number of motorized vehicles produced in the first 30 th years of the 21 st century is estimated to be much greater than the total of motorized vehicles throughout the20th century. Furthermore, these vehicles are distribute much more in developing countries, meaning that accidents and fatalities are greater in low and middle-income countries.
[1] [2] According to WHO, 2013, in the year 2010 at least 1.24 million accident victims have died and 20 to 50 million were injured. Casualties from traffic accidents became the fifth largest caused of death in 2013, and every hour in Indonesia there are 3-4 people who die due to traffic accidents, which is a much more than the death rate due to war in Afghanistan, there fore1-2 people per hour. The government is very aware of this complicated challenge, because traffic accidents cause more deaths to the employed age groups (productive age). The number of motorized vehicles in Indonesia continues to increase rapidly; in 2014, there were 114,209,260 units of motorized vehicles and in 2016, the number increased to 129,281,079 units. The government has made a breakthrough by publishing the INPRES No. 4 of 2013 concerning the 2011-2020 Road Safety Action Decade which coincided with the declaration of the National Road Safety Plan (RUNK) declared in 2011. To reduce accident fatalities, the Indonesian government carried out a coordination pattern of inter-sectoral synergy that is coordinated by BAPENAS. The sectors are divided into 5 pillars of safety, namely: Pillar Ministry of Public Works, in this case, the Director General of Highways is responsible for provided safe road infrastructure and is obliged to make improvements to accident-prone locations.
While the Directorate General of Land Transportation is responsible for harmonizing signs and marking of road functions, one of the 9 activities carried out is the Road Safety Audit. The target is reducing accident fatalities , such as the target of the Second Pillar by the Ministry of Public Works, it requires hard , fast work and intense coordination for all involved organizations. Therefore, there is a considerably large amount of roads that must be audited, one of which one is an audit of road signs and markers. One of the factors that increasingly promotes road safety audits is that traffic safety is no longer just based on driver safety, but rather a traffic system error (including road infrastructure) [3]. Supported by a greater responsibility now , is on the road operators, namely the activities is planning and regulation of the traffic system that is capable of creating quantitative indicators, monitoring and evaluating road safety deficiencies. [4] According to [5], the number of traffic signs on the Cipularang Toll Road is still lacking, especially warning and instruction signs that serve to direct and guide the flow of traffic, and the technical specifications for signs and markers are still lacking as well. Various types of signs and road markings on the Cipularang Toll Road meant to assist drivers in driving are laid out in accordance with the Republic of Indonesia Transportation Minister's Regulation Number PM 13 of 2014 concerning traffic signs and the Republic of Indonesia Transportation Minister's Regulation PM 34 Year 2014 concerning road markings. With the implementation of road safety audits , we are expected to be able to find out the effectiveness of road signs and markers as well as their suitability when compared to regulations set by the government and their influence as one of the causes of accidents . This is to achieve three aspects of road safety, namely forgiving Road Environment, self-explaining road, and self-regulation road

Problem Identification
The underlying problem is the number of accidents occurred in the Cipularang Toll Road, despite the fact that the Toll Road has been equipped by traffic signs and road markings to help the driver through the road. Road signs and markers must be able to provided clear information so that the driver can always be ready to anticipate various traffic situations while driving.

Scope of Problem
The problems discussed are limited to taking road inventory data only at km 96 to km 100 which is then compared to the regulation of the [7]concerning traffic signs and the regulation of the [8]concerning road markings and the [9] concerning road planning specifications.

Purpose of Research
Based on the background, the purpose of this study is to determine the effectiveness of signs and road markings on road conditions that are straight, downhill and uphill and to determine whether there is a significant differences between incorrect sign and marking placement with the accidents. Traffic Signs The regulation of the [7] concerning traffic signs states that traffic signs grouped based on their function, The images of traffic signs can be seen in Figure 1   The Various types of traffic signs that are according to the group, namely warning signs, prohibitory sign , Mandatory sign, Non-Expressway sign and additional sign, are describe in figure 1; 1a, 1b, 1c , 1d, 1e, and 1f respectively. The Warning signs, with a yellow base color, black borders, black symbols, letters, or numbers discribed on figure 1a. Warning signs are used to warn of possible hazards on the road or dangerous places on the road and inform about the nature of the hazard. Then the Prohibitory signs; with white base color, red borders,black symbols, letters, or numbers, and red words ilustrated on figure1b. Prohibitory signs that are used to express actions that are prohibited to road users dicribed on figure 1c. The signs with the following characteristics: blue base color, white borders, white coat, white letters and/or numbers; and white words namely The Mandatory signs, are used to declare orders that must be obeyed by road users, explained on figure 1d. And then Nonexpressway signs on figure 1e: to provide information to road users . The trademark characteristics are: green base color, white border, white symbols, white letters and/or numbers. Also the last on figure 1f , the additional Signs, are those provide additional information for road users.

Road Markings
The regulation of the [8], concerning the road markings defines, a road marking is a sign on the road surface or above the road surface which includes equipment or that signs form longitudinal lines, transverse lines, oblique lines, and symbols which serves to direct the flow of traffic and limit the area of interest in traffic. The image of markings categorized into fig. 2  . Also Refer to figure 2d, the Symbol markings, are road markings in the form of arrows, images, triangles, or writings that are used to repeat the purpose of traffic signs or to notify road users that can't be stated by traffic signs. And the Yellow box markings, are yellow rectangular road markings which function to prohibit vehicles from stopping in aspecified area according on figure 2e. Then the Last Other markings comprise of, a road crossing marking, no parking or stopping area marking, bike lane markings, markings of special bus lanes, motorcycle lane marking, tourism location entrance markings, alert markings, evacuation lane markings, school safe zone markings, also warning signs of intersections between rails and roads, are following figure 2f.
Uphill and downhill road conditions or vertical alignment is the intersection between the vertical plane and the axis of the road[10].Vertical alignment design greatly affects the volume of earthwork that will be carried out.

Method of Collecting Data
The method used for the data collection of this study is to conduct direct observation in the field (Road Inventory) using Go Pro video cameras and visual observations along the Km 96 to Km 100 section of the Cipularang Toll Road. The distribution and collection of factual data about road signs and markings in the Cipularang toll road is completed by signs and markings road inventories to obtain road characteristic data. Data of road characteristics in Cipularang Toll Road are collected by taking data every 50m length section , from Km 96 to Km 100. From the data on road characteristics we can find out the geometric conditions of the road at that location and the road signs and markings that are will be present. Then from the data obtained from the road characteristics, is created by a table form to make it easier to read the data .

Data Analysis Method
Comparisons of actual traffic signs and road markings on field observations (road inventory), done along the Km 96 to Km 100 of the Cipularang Toll Road. The form of characteristic data which is the comprehensiveness of road markings and traffic signs as well as the road geometry variations have to accordance with the technical specifications of the Ministry of Public Works and the Ministry of Transportation.

Analysis and Discussion
The results of data collection of road characteristics are in the table form to make it easier to see signs and markings at each location. That can be seen in table below , there is many location which in actual conditions has a lack of signs on towards Jakarta which is not in accordance the Ministerial Regulation No. 13 of 2004, namely the absence of warning signs to warn the driver that road conditions are no longer flat but inclined downward.  Although in the actual conditions there is a shortage of signs on the many location in direction of Jakarta , a driver who passes Cipularang more than 4 times in 1 year don't have an effect on their driving comfort . This indicates , the driver does not fully attention to the presence of road signs or markings in any road conditions indicating that they are not aware of the lack of signs or markings in certain locations. [11]This became known when conducting direct interviews with drivers at the rest area of the Km 72 in the direction of Bandung and Km 97 in the direction of Jakarta.
From the factual conditions that exist in the Cipularang Toll , data from the table made to clearly road characteristics data, for example, the KM 97 + 35 in the direction from Jakarta to Bandung , on area accident-prone catagory, that even though the road conditions began to up hill, there were no warning signs indicating those road conditions. Based on [7] article 39 paragraph 2, before reaching potentially dangerous road section , there shall be a warning sign installedat least 80 meters for roads with a design speed of 60km/h to 80km/h and at least 100 meters for roads with design speeds of 80km/h to 100km/h. The absence of signs on Km 97 + 35, on area accident-prone catagory, is a form of violation of the regulation [7]. However, the results of processing the perception of the driver's data on the signs indicate a fairly good level of satisfaction. This indicates the driver does not fully pay attention to the presence of road signs or markings in any road conditions, that mean they are not aware of the lack of signs or markings in certain locations.

Conclusion
The conclusions drawn from the analysis results in this study are: From the table made based on the actual conditions on the Cipularang Toll Road, that can be concluded , a lack of warning signs on KM 97 + 35direction to Bandung , that indicate the road is began uphill. This is not in accordance with regulation [6] Article 39 Paragraph 2 concerning the placement of warning signs that require warning signs to be installed at least 80 meters for roads with a design speed of 60 km/h to 80 km/hour and at least 100 meters for roads with a design speed of 80 km/h to 100 km/hour.
Comparisons between the exist Road Inventory data and driver's perception show that even though in actual conditions there is a lack of signs on the Km 97 + 35 in direction of Jakarta 9 downhill), the driver who passes Cipularang more than 4 th times a year doesn't have their driving comfort affected, so even if there are locations that lack warning signs, the driver's perception still showed a good level of satisfaction. It can be concluded that the unavailability of signs doesn't affect the driver in reaching their