Mini Kirsch Edge Detection and Its Sharpening Effect

Received Jul 6, 2020 Revised Mar 12, 2021 Accepted Mar 17, 2021 In computer vision, edge detection is a crucial step in identifying the objects’ boundaries in an image. The existing edge detection methods function in either spatial domain or frequency domain, fail to outline the high continuity boundaries of the objects. In this work, we modified four-directional mini Kirsch edge detection kernels which enable full directional edge detection. We also introduced the novel involvement of the proposed method in image sharpening by adding the resulting edge map onto the original input image to enhance the edge details in the image. From the edge detection performance tests, our proposed method acquired the highest true edge pixels and true nonedge pixels detection, yielding the highest accuracy among all the comparing methods. Moreover, the sharpening effect offered by our proposed framework could achieve a more favorable visual appearance with a competitive score of peak signal-to-noise ratio and structural similarity index value compared to the most widely used unsharp masking and Laplacian of Gaussian sharpening methods. The edges of the sharpened image are further enhanced could potentially contribute to better boundary tracking and higher segmentation accuracy.

INTRODUCTION
Edges in a digital image are the pixels where their gray level intensity changes abruptly compared to their local neighbouring pixels. In the application of image processing, machine vision, and computer vision, edge detection is one of the crucial steps in pre-processing stages for finding the boundaries of objects within an image, for instance detecting local discontinuities in pixels intensity or brightness for boundaries extraction [1]. Edge detection is widely implemented in the application of car's license plate detection [2], human face recognition through iris localization for eye tracking [3], synthetic aperture radar images to detect edges of ships, aircraft, terrain, meteorological forms and mobile vehicles [4], agricultural plant leaves recognition [5], and dehaze or deblurring method [6]. Furthermore, biomedical image, i.e. magnetic resonance image (MRI) [1] [7], ultrasound, X-ray, and computerized tomography (CT) scan segmentation is a challenging task with the existence of speckle noise, its poor luminance and indistinctive contrast [8] which cause difficulties in identifying the edges [9][10] [11] [12].
The existing edge detection algorithms that are implemented in the spatial domain can be generally divided into two groups: gradient-based and Gaussian-based [13]. Examples of gradient-based edge operators are Sobel, Prewitt, Roberts, and Canny. Meanwhile, Laplacian of Gaussian (LoG) is categorized under the gaussian-based method, refer to Table 1. Gradient-based operators are filters with different kernel sizes which are then convolved with the original input image to obtain its image gradient. Gaussian-based methods are also known as zero-crossing or second-order derivative methods. The Canny operator is well-known for its outstanding performance as it undergoes smoothing operation for noise reduction, gradient magnitude computation as gradient operators, thresholding to keep strong edges while removing the weak one and finally non-maximum suppression for edge thinning process [14]. Sharpening is another crucial stage to boost the edge detection performance. It helps to emphasize the tiny details and boundaries of objects in an image [15]. Generally, it is carried out at the post-processing stage to enhance the overall image visuality and quality. It can also work together with the edge detection operators to result in a better edge segmentation. There are two well-known and most popular sharpening algorithms: Gaussian high pass filter and unsharp masking method.
Existing gradient-based edge detectors benefit in terms of simplicity and gradient estimation in several directions. On the other hand, the approximation could be affected by the level of noise and the magnitude of edges [16]. Overall, these operators cannot result in accurate edge detection especially those thin and faint edges. Mini Kirsch edge detection method with four-directional filters was proposed to detect edges of an object from all angles. Inherit from the original Kirsch edge detection method with eight-directional filters, the maximum edge magnitude was measured across all gradient magnitude in all directions. The normalized edge detection result was then added to the original input test image for the sharpening effect. The edge detection performance of the proposed method was compared with Sobel, Prewitt, Roberts, LoG, Canny, and Kirsch algorithms while its sharpening effect was examined together with unsharp masking and LoG.

RELATED WORKS
For gradient-based or first-order methods, edge detection is carried out by convolving the kernels along the input image to measure its gradient magnitude. Meanwhile, gaussian-based or second-order approaches capture local maxima of the image gradient, the process is also named as zero-crossings of the Laplacian operator. To yield a better edge detection result, a smoothing process, typically Gaussian smoothing and anisotropic diffusion, will be done in pre-processing step [16] [17]. As reported by Iskandar, Gaussian filter kernel size is critically important to result in different levels of the blurred image. Greater filter size could expect a blurrier output image. Therefore, the non-local means (NLM) filter is suggested to replace the Gaussian smoothing filter as it evaluates the similarity of the pixel with nearby pixels, taking into consideration not only their intensity difference but also from the whole spatial neighbourhood [1]. Another published framework has proven that the canny method could be further improved with an appropriate Gaussian filter for effective white noise reduction by extending its image gradient magnitude with the Sobel algorithm [16]. A threshold is normally applied after the computation of edge magnitude to decide whether an edge pixel is to be kept. More edges will be detected when the threshold set is low and vice versa. Assisted by adaptive thresholding, edges of human iris and pupil resulted from Sobel operators and three points are selected randomly to generate a circle for human eye localization [3]. With a high threshold value, we might miss subtle edges, or most of the time output undesired edge fragments. Statistically, it is being proven that the combination of search spaces could help in faint edge detection, the authors also use two different algorithms for both straight and curved edge detection [18]. To detect sharp changes in image field depth, a multi-scale filter window is suggested, such as using 5 x 5 window for regions with larger field depth transformation while 15 x 15 window size for regions with smaller field depth transformation [6].
The author implements eight-directional edge operators to field the depth of the image and apply a guide filter to enhance the edges in the image, later the texture difference between the original image and the edge-enhanced image is measured to dehaze the foggy image. Images are taken under different light intensities, the objects in the images could be exposed to the light unevenly which affects the edge detection accuracy severely. Some faded edges might appear to be too soft for detection. Therefore, log Sobel is proposed to process optical density to remove the uneven illumination intensity. The algorithm is backed with photography technology utilizing the proportional relationships between the logarithm of light intensity and the logarithm of the object's reflectivity [19].
Edge detection in a colour image with anisotropic morphological directional derivative matrix (AMDD) is more resistant towards noises like Gaussian and impulsive noises [20]. More colour image existing edge detection algorithms are color Canny [21], PCA-GOHD [22], RCMG-MM [23] and ColorED [24]. Edge detection in colour images consumes a longer time as the detection is required to be carried out in every colour channel. Another interesting novel finding by Delgado shows that edge detection could be done without a fixed kernel. A non-local fractional mask is designed, the parameters take account of visual perception and its statistical analysis, allowing it to extract more fine image details from overall spatial information [25]. Besides, there is a study on edge detection in the frequency domain with the framework that connects Ridgelet transform, Bezier curve, and Sobel operator. Ridgelet transforms threshold the existing noises while Bezier curve can perceive the details to avoid huge deviation of data [4].
Image sharpening is defined as an acutance improvement that enhances the apparent quality (edge information, contours, texture, etc.) without changing the content in the image. Furthermore, a good sharpening algorithm aims to improve the image details without adding unwanted and excessive artifacts that can decrease the image quality and increase the difficulties in image segmentation later. In recent years, research on image sharpening has been growing actively due to its existing challenges, including the over-sharpening effect, amplification of latent noise, overshoot and others. Among the existing sharpening algorithms, various popular techniques are Laplacian filtering, unsharp masking (UM) [10], shock filtering and local neighbourhood operators [15] [26]. Zohair proposed a nimble filter, a single kernel to obtain specific spatial information from the input image and deduct it from the original image with certain weighting parameters to control the level of sharpness improved [15]. The spatial information is computed by measuring the mean of both horizontal and vertical shifts of the input image, then it generates a blurry image that highlights its edge structures. The method is inconvenient since the weight parameters must be manually pre-determined to yield a satisfactory result. Inspired by Zohair, Lin et al. suggested measuring the gradient norm value of every single pixel to be calculated and acts as a reference on determining the categorization of a particular pixel to the edge. Meanwhile, the edge can be identified by taking the turnover points from the second derivative of a curve produced previously. The proposed scheme successfully preserves the edges and amplification of noises [27]. UM is a classical image sharpening method, which is available in most consumer products like cameras and photo editing tools (i.e. Adobe Photoshop). An unsharp mask, in other words, a smoothed image from the corresponding image, is obtained by conducting a smoothing process of the original image with a Gaussian filter. The unsharp mask includes high-frequency information and edge details. Later, the mask is added to the original image which potentially emphasizes the noise. Chen et al. claimed that reverse anisotropic diffusion (RAD)-based technique to be the successor of UM method as not only edges are preserved, but also the salient details in the image. The original anisotropic diffusion model has a characteristic of smoothing images while preserving the edges along the boundary of the objects. Different case to RAD-based UM, reverse anisotropic diffusion tends to smooth the edges while keeping noise in the image [10].
Another innovative work to extend Hausdorff derivatives to image sharpening. The Hausdorff derivative Laplace equation (HDLO) is computed through Hausdorff fractal distance [32]. A hybrid deblurring strategy introduced by Chang et al. tends to process strong edges and weak edges separately. The author applied a patch-based method to sharpen the existing strong edges while using bilateral filtering to remove the noise and smooth edges [26]. Canny edge detection provides double threshold values, high and low thresholds, which are critically important to filter out the weak edges while retaining the strong edges. Therefore, the performance of the proposed model is heavily dependent on the choice of both threshold values.

Software and Computer Specification
All the algorithms and performance tests mentioned are run through MATLAB 2019a on a laptop hosting Intel Core i7-6700HQ quad-core with hyper-threading clocked at 2.6 -3.5GHz, 8GB DDR4 RAM, and NVIDIA GTX960M (4GB GDDR5 RAM).

Proposed Mini Kirsch Detection Kernels
The original Kirsch operator is a non-linear edge detector that takes a single 3 x 3 kernel and rotates it in 45 degrees to form a total of eight-directional operators. The directional kernels operate through eight different directions: north, northwest, west, southwest, south, southeast, east and northeast. Every kernel will produce a different image gradient map through simple convolutional processes. Later, the maximum magnitude is chosen among the eight gradient maps and finally results in a complete image gradient map. Overall, Kirsch operators manage to delineate the structure of objects in the image. Inspired by Kirsch operators, we proposed to modify the operators by reducing the kernel size to 2 x 2 which could achieve satisfying performance with only four-directional kernels Similarly, the final edge magnitude map is obtained through the computation of maximum gradient magnitude across all four image gradient maps with the equation: Where indicates the input image size and ℎ represents the parts of pixel location to be convolved with 2 × 2 kernels ( ) across all the pixels in the images. The brightness of a particular pixel is measured based on its nearby pixels. If all the pixel intensities within the convolution window are similar, we could say there is no edge point at that particular location. For instances, kernel (1) manages to identify the edges that fall in the northwest, west and also north direction. Meanwhile, kernel (2) can detect edges pointing southwest, west and south. With these four aforementioned kernels, we can expect the edges detected to be covered throughout all the directions.
Referring to Figure 1, the grayscale image of an air jet is input into the system. Four gradient magnitude maps are yielded from the convolution between the kernels with the input image. Every kernel detects its corresponding edges respectively and therefore outputs its gradient map differently. After combining the four resulting gradient maps by taking maximum magnitude across the maps, the output pixels might contain negative and non-integer values, and therefore normalization should be done to normalize the pixel values ranging from 0 to 255.
The normalized map with edge enhancing details is added back to the original input image to sharpen the image. However, adding two images might cause the image to be brighter as the intensities might be shifted towards the brighter side of the histogram. At the same time, the image contrast could be badly influenced. Hence, contrast enhancement is critically important to recover the visuality and the illuminance of the output image. In this paper, linear contrast enhancement is applied to stretch the intensity distribution to enhance the overall image quality. We can observe a better edge presentation in the output air-jet image after the proposed sharpening method.

Sample Images and Edge Detection
All the images in Table 2 and Figure 2, and their ground truth edge maps tested in the edge detection section were obtained from Berkeley Segmentation Dataset [33]. The mammogram in Figure 3 was obtained from [34], aneurysm image in Figure 4 was retrieved from [35], brain MR image in Figure 5-1a from [36], funduscopic images ( Figure 5-2a) was provided by the Department of Ophthalmology at school of Medical Sciences, Universiti Sains Malaysia and 3-Tesla knee MR scan was retrieved from [37]. Meanwhile, images in Figure 6 were obtained from [38], [39], and [40].

Edge Detection Comparative Test
Ground truth image is the standard for benchmarking and an algorithm performance evaluation. It is often used to measure the deviation of our output from the actual result. We selected five images downloaded from Berkeley Segmentation Dataset to be tested with our proposed algorithm, Sobel, Prewitt, LoG, Roberts, and Kirsch methods. The five images' IDs were 12003 (starfish), 300091 (surfing by a man), 35010 (butterfly with complex background), 314016 (dolphin) and 242078 (outdoor folding umbrellas). Edge maps yielded from different methods will be compared with the ground truth edge map provided by the database. The edge detection methods were evaluated in several forms: the number of true positive, true negative edges detected, the accuracy of detection and F measure.
True positive (TP) is the number of edge pixels that are correctly detected. Meanwhile, true negative (TN) is the amount of non-edge pixels that are correctly identified. Let O be the output image while I be our grayscale input image, TP can be expressed as: While TN is the intersection of both complements of input and output images.
Accuracy measures the overall performance of the edge detection algorithms. Taking account for both TN and TP, the equation is illustrated as follow: The higher the accuracy achieved by an algorithm, the better the edge detector's performance to be expected. A low accuracy rate could be expected if the total of TP and TN edge and non-edge pixels are detected by the algorithm is small. F measure is a performance test that combines precision and recalls to compute the final score that is not lower than 0. The higher the F measure value, the better the predictive ability of a particular edge detection method. In an ideal case of edge detection, F-score would be 1. In reality, F-score that is nearer to 1 is considered to be having decent performance.

Sharpened Image Quality Test
The final edge map obtained by computation of maximum gradient magnitude, produced by the proposed four convolutional kernels with the input image, possibly contains negative pixel values. Normalization is carried out, in this case, to normalize the whole pixel values in the image ranging from 0 to 255, and thus the final pixel values of the edge map will be in positive values. Adding the normalized edge map to the original input image potentially increases the artifacts and noises of the output image. Therefore, we evaluate the sharpened image quality with a peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM).
The PSNR indicates the ratio between the maximum possible pixel intensity value of the input image and the mean squares of the errors between the input and the sharpened image. A high PSNR value indicates good details preservation and noise resistance offered by the sharpening algorithm. Let Considering our input image has a size of M rows and N columns. Mean squared error, MSE can examine the changes of the sharpened image from its original input image and can be denoted as Abrupt changes, for instance, noise and artifacts addition could result in a higher MSE in other words, the sharpened image is severely degraded after the sharpening step. SSIM index is a predictive method to measure the similarity between sharpened images and the input image. Structural information in an image is the strong dependencies of the pixels that delineate the structure of objects in an image. SSIM examines the structural information degradation in the image. SSIM formula includes three comparison measurements which are luminance ( ), contrast ( ) and structure ( ). For a simpler representation, let = , ℎ = .
Luminance comparison, ( , ) = 2 + 1 2 + 2 + 1 (11) Contrast comparison, ( , ) =  (13) where , , , and represents the local means, standard deviations and also the cross-variance for both original input image and edge-enhanced image respectively. Moreover, 1 , 2 , 3 ∈ ℝ + are the regularization constants for the three comparisons above. The three comparisons are later combined as follow: where α, β and γ are the weights and equal to one in this case. Finally, SSIM can be written as A high SSIM value indicates that the final sharpened image and the original input image are quantitatively similar. In simple words, the addition of an edge map into our input image does not change the structures of the objects in the image.

Edge Detection Performance Test
A high accuracy edge detection rate indicates the successful classification of true edge pixels or true positives and true non-edge pixels or true negatives. From Table 2, the proposed method yielded the highest accuracy in matching the ground truth edge map with the detected edges. By applying a threshold ranging from 50 to 55 to remove the existing noises, the accuracy rates obtained in the starfish image were 0.8287, 0.9008 in surfing image, 0.7710 in butterfly image, 0.8121 for dolphin image, and 0.8207 for outdoor umbrellas image. Notably, the Kirsch method gave the highest F value yet lower accuracy in most cases. In other words, Kirsch managed to detect more TP edges but it failed to classify non-edge pixels.
Roberts maintained its accuracy in detecting edges while Sobel and Prewitt methods often miss approximate the edges in the images, as reported in [19]. Mini Kirsch kernel found the maximum edge magnitude in several predetermined directions by rotating a 4 x 4 kernel. Inherit from the compass feature as reported in Kirsch operator [31], the proposed method could delineate the boundaries of the sample images with high continuity edge pixels as shown in Figure 2.
In biomedical image processing, proper edge detection in medical images is crucial, typically for human organ recognition or cancer cell detection [41]. However, medical images are contaminated with speckle noises, low background illuminance and resolution which increase the difficulties in identifying the region of interest from nearby tissues [7]. The contribution of the proposed method in biomedical image processing was further discussed in parts 4.1.1 and 4.1.2.

Edge detection on Mammograms
A mammogram is an X-ray image of the breast captured for breast cancer detection. Breast cancer detection procedures can be divided into a few stages, namely region of interest extraction, identification of suspicious tissues and tissue classification [42]. A good edge detection algorithm is vital in the region of interest extraction stage which can result in good identification and segmentation [5] [1].
As can be seen, Figure 3 showed that the proposed method managed to detect the lump structure with high edge pixel continuity compared to other methods. Accurate edge detection could visually assist physicians in making decisions and classification.

Edge detection on aneurysms images
Aneurysms refer to abnormal arterial dilatation and weakening of an artery wall, that can cause rupture and life-threatening internal bleeding [25]. Therefore, early prediction of rupture is needed yet the local inhomogeneous contrast makes the edge detection becomes more challenging [43]. The canny method outperformed in tracking strong edge pixels compared to other gradient-based methods. Meanwhile, the second-order derivative failed to remove the noises and its amplification eventually deteriorated the image quality which could be seen in Figure 4-d. Our proposed method could identify the edges of vessel trees much better than the other methods. The resulting edge map clearly distinguished the vessel regions from the background.

Sharpening Image Quality Analysis
As can be seen in Figure 5 and Figure 6, the image quality difference between the original input image and the sharpened image was hard to be differentiated. Hence, we evaluated the sharpened image with PSNR and SSIM test. Adding an edge map onto the original map might cause noise amplification as well as introduced undesired artifacts which could greatly degrade the quality of the image [27]. Data collected in Table 3 reported that the proposed method was considerably competitive with unsharp masking and Laplacian of Gaussian filter. A high PSNR value indicates good details preservation while avoiding latent noise amplification [1]. Medical images consist of a massive amount of unwanted noises that increase the difficulty in retaining its algorithm precision [33]. Therefore, sharpened medical images like brain MRI, knee MRI and funduscopic image resulted in a lower PSNR value than another three tested images.
Inspired by Laplacian of Gaussian method [13], a light gray edge map was added onto the image for the sharpening process. However, the process potentially shifted the image histogram to be left-skewed. Biased histogram resulted in indistinctive deviation foreground and background identification [12]. Therefore, we applied linear contrast enhancement to recover the contrast of the output image to increase its visibility. The proposed method could obtain the highest SSIM value for funduscopic image and knee MRI scan, referring a piece of good structural information was preserved.

Edge Detection After Sharpening
In Figure 7, the edge detection of the image was conducted with the Canny method to examine the enhanced edge information done through our method. More window edges were detected after sharpened by the proposed method.
Diabetic retinopathy, caused by diabetes, could damage the blood vessels leading to vision loss [44]. Blood vessel tracking is essential to detect the changes in vessel pattern to find the abnormalities among the vessel. The existing gradient-based method and gaussian-based method are inadequate to delineate a good edge map to achieve the objective [45]. Having the funduscopic image sharpened with the proposed method ( Figure  8-c), more vanishing edges image could be detected as in Figure 8

Separating Regions with Strong Edges
In biomedical image processing, observed edges are important to offer a good segmentation for region-based methods, for instance, active contour [46] and region growing [47]. In Figure 9-(a to c), oversegmentation often occurs when the region of interest has homogeneous intensity distribution with neighbouring tissues. In region growing, a seed point is placed within an interested region. The region continues to grow by adding nearby similar pixels and iteratively merges the other region according to their homogeneity. Edge refers to the discontinuities in pixels intensity which is commonly used as stopping criteria in regionbased method [7]. The proposed edge sharpening framework contributed to image acutance improvement that can enhance the apparent quality while retaining the structural information. With sharpened edges, the regionbased method would stop expanding its region across over the clear boundaries that can be seen in Figure 9-(d to f). In short, the proposed kernel could function as a rotating gradient-based edge detector with a good sense of edge detection compared to other methods. The algorithm was further developed into a competitive edge map sharpening framework. However, the kernel was heavily dependent on noise existent as reported in Table 3. The sharpening frameworks possibly amplified the existing noises in medical images. We suggested that implements of smoothing filters at the pre-processing stage can overcome the claimed problem. Edgepreserving filters [17] such as anisotropic diffusion filter, bilateral filter and non-local mean filter were recommended. On the other hand, mean filter and median filter should be avoided in this case as it inclined to washout the edges which increased the difficulties in edge detection.

CONCLUSION
Edge detection aims to find the boundaries of objects in the image. Meanwhile, image sharpening is an initiative to improve its apparent quality without altering structures in the image. Existing edge detection methods detect fragmental edges where the discontinuity makes the further segmentation to be more challenging. Therefore, we proposed a four-directional mini Kirsch edge detection method that inherited the characteristics of the eight-directional Kirsch compass edge detector and is sufficient to detect edges from different directions. From the edge detection performance analysis, the proposed method outperformed in its accuracy in detecting true edge pixels and true non-edge pixels. Besides, it yielded a satisfactory result in the F measure which mainly focused on the effectiveness of true edge detection. Despite the ability of convincing edge detection by our proposed method, we also introduced the novel involvement of the sharpening method by adding the edge map produced by our methods onto the original image. The quality of the sharpened image was well preserved with its strength of noise removal and with its overall structures retained. However, our proposed method has drawback in processing medical images which are full of speckle and Rician noises. A threshold was not enough to remove the noises thus we suggested to add another smoothing process using edge-preserving filters such as anisotropic diffusion filter, non-local mean filter, or bilateral filter before edge detection. In a nutshell, the sharpened image with the proposed method managed to increase its acutance information, and therefore a better continuity edge map could be observed after the sharpening process.


Social Dimensions in CPS & IoT based Automated Production Systems

Since the 1970s, the application of microprocessor in industrial machinery and the development of computer systems have transformed the manufacturing landscape. The rapid integration and automation of production systems have outpaced the development of suitable human design criteria, creating a deepening gap between humans and systems in which human was seen as an important source of errors and disruptions. Today the situation seems different: the scientific and public debate about the concept of Industry 4.0 has raised the awareness about the central role humans have to play in manufacturing systems to the design of which they must be considered from the very beginning. The future of industrial systems, as represented by Industry 4.0, will rely on the convergence of several research fields such as Intelligent Manufacturing Systems (IMS), Cyber-Physical Systems (CPS), Internet of Things (IoT), but also socio-technical fields such as social approaches within technical systems. This article deals with different human social dimensions associated with CPS and IoT and focuses on their conceptual evolution regarding automated production systems’ sociability, notably by bringing humans back in the loop. Hereby, this paper aims to take stock of current research trends to show the importance of integrating human operators as a part of a socio-technical system based autonomous and intelligent products or resources. Consequently, different models of sociability as way to integrate human into the broad sense and/or the development of future automated production systems have been identified from the literature and analysed.

Introduction -Ground-breaking changes in Industry worldwide
Today, the German initiative Industrie 4.0 1 [1], along with many other national sociopolitical programs such as 'Industrie du future' in France 2 , 'High Value Manufacturing Catapult' (HVMC) in the United Kingdom 3 , 'Made in China 2020' 4 , 'Manufacturing USA' 5 , etc. are fostering the digitisation of industry and are taken as references for the development of new manufacturing systems. The strong stakes associated to their simultaneous and worldwide emergence make them the foundations of the 4 th industrial revolution as it is already considered by experts.
Basically, the technical innovations related to the vision of Industry 4.0 implies the wide-spread adoption of Cyber-Physical Systems (CPS), the integration of products, smart factories, and the introduction of value chains into global business networks [2,4,60,64].
The second contribution figures out, where automation processes are going on and how humans are integrated within these systems. Because ongoing process of automation belongs to the inner logic of industry, this aspect becomes explicit in the following paper, which is structured as follow. Section 2 will detail some fundamentals concerning CPS and IoT paradigms, and the role to be given to them within these automated production systems. Section 3 will discuss different models of sociability defined as the ability to interact with others including human-machine, machine-machine, human-human interactions, to illustrate the purpose of section 2. Due to these complex developments, the hypothesis implicitly is raised that the engineers and developers intend automation on different process levels. Section 4 will conclude by raising open questions concerning work automatization and questioning how automation is today changing work quality.

Fundamentals on CPS and IoT -an insight
The notions of CPS and IoT are generally recognized as the main pillar of Industry 4.0. [4,5] Decades ago, the debate on Computer Integrated Manufacturing (CIM) systems referred intensively to new technological and social dimensions with huge societal impact [15]. However, recent associated technical integration systems encompass this debate with different focus [16,19,26,31,33]. Technological dimensions are related to the connection between the technical systems while sociological dimension is related to communication and interfacing human and technical systems. In both cases, integration has to consider the following: a) machine-machine interaction, b) human-machine interaction and c) human-human interaction.
The introduction of CIM systems in the 1980s had fundamental effects on the organizational level of work. On the one side, digitization of work processes created a vision about the complete automation of factories without personnel [15]. Yet, on the other side, the high level of technical standards created a discourse about the rising and dependent scope of human actions within working processes [2].
We call 'restrictive' a work organization based on a strict human-machine interaction in delimited workspace, dependent on technical parameters and on rational production lines. This organization disappears while considering the support of qualified and responsible employees, which should be actively involved into the production processes. It must be noted that such flexible work organization is more complex to design than ones restricted to the mechanistic and hierarchical principles of management. The participation of the employees into these processes should imply the introduction of tacit knowledge, planning and operation, group work, as well as decision-making processes.
These criteria are again at stake when the emergent vision on the new automated production systems designed with CPS and IoT are debated [5,34,42,55,57,60,64]. Observations have already shown that these distributed internet-based systems bring path dependencies that may restrict the possibilities for alternative work organization models by automation [25,27,30]. If human factor is not included simultaneously with the technological factor into the design process, there is little space for 're-automation' regarding human interference into the work processes [19].
Due to the wide range of its potential applications, this concept of CPS enjoys great popularity in the scientific world although it is rather recent (enunciated by Lee in 2006) [22]. However, popularity and novelty make it a concept whose definition and scope are rather blurred. It is also often associated with the one of IoT, appeared a little earlier, in the 2000s [18].
According to Bril El-Haouzi [19] and Bordel et al. [20], preferences in the use of the terms CPS and IoT are observed from one scientific community to another or from one geographical area to another. Thus, CPS will be preferred to IoT in mechatronics and IoT in computer science communities. The term CPS will also be found more often on the American continent than on the European or Asian ones, where IoT will be preferred [19,20]. Yet, those two concepts are fundamentally different and need to be differentiated.
The popularity of the IoT concept have provided in literature with many definitions. Notably, Madakam and colleagues made a literature review in 2015, which led them to formulate the following definition: "an open and comprehensive network of intelligent objects that have the capacity to auto-organize, share information, data, resources, reacting and acting in face of situations and changes in the environment" [21]. Concerning CPS, Lee initially enunciated the concept as being "integration of computation with physical processes" where "embedded computers and networks monitor and control the physical processes, usually with feedback loops where physical processes affect computations and vice versa" [22]. It can be noted that this definition stays consistent with the one fostered more recently by Monostori "systems of collaborating computational entities which are in intensive connection with the surrounding physical world and its on-going processes" [23].
These first definitions can be characterized as techno-centred since only technical aspects of CPS and IoT are exposed, while human is not evoked. Hence a system can be considered as being composed of both objects and their cyber representations. In other words, a system can be seen as organized along two axes: he first one representing the physical world; the second one being representing the cyber world. In the one hand, IoT would correspond to the horizontal connectivity and synchronization between physical or cyber objects, performed thanks to Internet data exchange protocols-based technologies (such as TCP/IP). On the other hand, CPS would correspond to the vertical connectivity and synchronization between objects and their cyber representation, performed thanks to cloud and sensors-based technologies [24,25,63,64,67] (Fig.1).

Figure 1. CPS & IoT [24]
Therefore, CPS and IoT can be seen as the two main enabling paradigms for Cyber-Physical systems' networking, and two of the main pillars for Industry 4.0. This representation seems well-suited to describe machine-machine interaction. But as expresses here above, human-human interaction and human-machine interaction (i.e.: HSI) are equally important regarding Industry 4.0 [2,19].
These aspects have nonetheless been tackled in for both IoT and CPS in the last decades. Chart 1 shows the results for the searches (S1): "Cyber Physical Systems" AND (Human OR Social OR Anthropocentric) and (S2): "Internet of Things" AND (Human OR Social OR Anthropocentric). This search was performed with the ScienceDirect scientific database for its practicality of use. No other databased was queried, for the purpose of this search is to provide an overview of the scientific interest for the subject, and not an exhaustive analysis. To get relevant results, the search has been restricted to articles (Research & Reviews), presenting the terms in their title, abstract or keywords. Hence, searches are targeting the articles not only mentioning them, but for which they are the main subject (  The interest shown for this research area has been overall constantly rising for the last decade. The decrease observed for 2020/2021 being due to update delay in database (for instance, results for CPS in 2020 have raised from 45 to 50 between April and May 2021). In a prior paper, Valette et al. have already proposed a study upon the humanintegration the evolution of CPS and IoT paradigms regarding human, anthropocentric and social characters [25].
What will be deepened in this paper is the social character, often associated with the development of systems and of their architectures. As Moniz and Krings mentioned, the application of new robotic systems in the manufacturing shop-floor level is widely done without the reference of «real» social implications. Due to the cognitive and perceptual workload for new robot operators in complex and automated working systems, the social dimension is currently defined by developers with the focus on the issues of security and, eventually, qualification. But the social dimension remains in a limbo of the technical debate about interaction systems [26].
The analysis about the impact of automation on work processes is closely connected with the knowledge about social issues at work level. If the term 'social' can be seen as the most generic to express human consideration for production systems design in literature, it is however not unambiguous nor necessarily the mark of an attention paid to the human being. We can especially cite its use in Multi Agent field, where any interaction between two agents, human or not, is called social.
The 1998's work from Sycara can notably be quoted, where 'sociability' is referring to the fact "that an agent is capable of interacting in a peer-to-peer manner with other agents or humans." [27], along with more recent works from Nguyen and Katarzyniak, establishing as social interactions "all acts, actions and practices that involve more than two agents and affect or take account of other agents' activities, experiences or knowledge states" [28]. Hence, social integration models/approaches can be divided into the three types: social interactions based on peer-to-peer communication interfaces, Social-Network Services based approach as a media for social interaction and Human-inspired social relationships-based sociability model, detailed in the following sections.

Social interactions based on peer-to-peer communication interfaces
The "classic" approach, mostly found in the Multi-Agent Systems (MAS) study field, defines as social any agent able to interact with another one, whether it is artefactual or natural. With Human-integration purpose, the developments we would classify here mainly aims to human physical enhancement, by technological means. Hence, Sowe et al. define a Cyber-Physical Human System as an "interconnected systems (computers, cyber physical devices, and people) "talking" to each other across space and time, and allowing other Preprints (www.preprints.org) | NOT PEER-REVIEWED | Posted: 26 July 2021 doi:10.20944/preprints202107.0557.v1 systems, devices, and data streams to connect and disconnect" [29]. This definition is found in the work of (Schirner et al. 2013), who developed the 'Human-in-The-Loop Cyber-Physical Systems' (HiTLCPS). This concept consists of an embedded system improving the ability of a human being to interact with his physical environment (Fig. 3). The "loop" is made up of a human, an embedded system, and their environment. Beyond the very concrete translation of the integration of the human factor into systems, they provide here a solution where the system is presented as a physical extension of the human being, via a digital interface.

Figure 3. HiTLCPS [30]
With the development of the 'Anthropocentric Cyber-Physical Systems' (ACPS) by Pirvu and colleagues [31], defined as a reference architecture integrating the three physical, cyber / IT and human components (Fig. 4), the search for the integration of the human factor is taken further. The authors present it as an integrated, social, local, irreversible, adaptive, and autonomous system, in line with the continuity of Cyber-Physical Social Systems (SCPS) and Cyber-Physical Social Systems (CPSS). The ACPS is presented as an architecture "where the humans are not just interactants with a CPS, but elements of the system affecting its lifetime behaviour" [31]. However, unlike previous contributions offering concrete applications, this one push human integration further but remains, therefore, very conceptual. The most recent development of these approaches until now is the 'Social Human-In-The-Loop Cyber-Physical Production System' (Social-HITL-CPPS) of [32]. In this paper, the interpretation of human agent's behaviour and its coordination with other agents are identified as the two main challenges of the integration of humans in social (and not just industrial) environments.
To meet these challenges, a three-layer architecture has been proposed. This architecture connects, on the one hand, human users to the cyber part via user interfaces, 7 of 15 and on the other hand, the physical parts (i.e., non-human agents and the environment) to the cyber part via a network (Fig. 5). Under these conditions, the social interaction with automated systems (i.e.: CPS) integrates several features, like human perception, sensing, haptic interaction, or communication. When people interact with robots and other CPS it must exist some degree of awareness of the human in the loop, in terms of sensing abilities and/or interfaces and abilities of the technical system to interact and communicate with people [33,62,63].
The fact that robots with such 'cognitive' abilities are introduced in a working environment means the relation between humans and these machines also evokes the relation between co-workers and the human resource management strategies in a company [34]. Once the socialisation (the ability to interact with others, or to socialize) of the automated production systems is achieved, several scientific questions that can only be resolved through further research remain open. In particular, as the capacities and abilities of humans and intelligent systems are not similar; thus, the recognition of differences is most relevant for a balanced architecture, with a better allocation of competencies and complementarities based on new technologies as Augmented Reality, IIoT [66,70].

Social-Network Services based approach as a media for social interaction
This second approach is based on the use 'Social Network Services' (SNS) type applications (e.g., Facebook, Twitter, Instagram, etc.) as a media for social interaction between human-human, machine-human or machine-machine. Between 1995 and 2020, a consequent raise of internet users and internet-connected devices has been observed [35,36](« Internet World Stats » 2020) (Fig.6).
Nomadic communicating objects such as laptops, smartphones, and tablets, have become omnipresent in our everyday life. SNS, whose development have been fostered by these devices, have been defined by [37] as "web-based services that allow individuals to (1) construct a public or semi-public profile within a bounded system, (2) articulate a list of other users with whom they share a connection, and (3) view and traverse their list of connections and those made by others within the system". Preprints (www.preprints.org) | NOT PEER-REVIEWED | Posted: 26 July 2021 doi:10.20944/preprints202107.0557.v1 Figure 6. Internet users growth from 1995 to 2020 [35,36] First consequence of this raise has been the generation of a huge data among, posing data structuration issues, leading Guinard et al. to the idea of using the structures of existing SNS to connect IoT devices into a 'Social Web of Things' (SWoT) [40] (Fig. 6). The SNS's ability to collect and process data to support the creation or maintenance of social relationships between their users, is there seen as a new way to structure data exchanges within a network of intelligent connected objects (i.e., artefact agents).
Today, this idea is fuelling the development of resilient data collection and sharing methods aiming to improve reputation, trust, and security between IoT devices [38,39,68] (Fig. 8). These methods are based on Graphs, to structure data-connection between devices, Degree distribution to quantify a node's solicitation, and Local Clustering Coefficient to group interlinked nodes as network clusters. Combining these methods to friendship-like relationships ultimately leading to a "social" SNS-based approach.  But Data structuration is not the only use that has been found for SNSs into systems' design. Social networking can equally be used as a way to organise manufacturing systems into distributed Dynamic Resource Communities (DRC) as a "new cyber-physicalsocial-connected and service-oriented manufacturing paradigm" [41]. This Social Manufacturing (SocialM) approach is based on the use of both socialized resources, social media, and social community inspired self-organization for resources (Fig. 9).
Resource agents (here named Production Service Providers or PSPs) are interacting with each other through a global social relationship network (i.e.: the SNS), enabling them to self-organize into these distributed DRC, aimed to bring resiliency and flexibility to production systems. Figure 9. Logic framework of SocialM [41] The notion of social manufacturing can be found again on the work of Romero et al on Social Factory Architecture based on Social Networking Services and Production Scenarios. They define the Social Operator 4.0 as a "type of Operator 4.0 that uses smart wearable solutions together with advanced human-machine interaction (HMI) technologies to cooperate with other 'social operators', 'social machines' and 'social software systems' to communicate and exchange information for mutual benefit and align/alter activities as well as share resources so that more efficient results can be achieved at the smart and social factory of Industry 4.0" [42].
This will increase the complexity of data management and recurrent amalgamation of roles (as today occurs with domestic bots of IOS or Google, as examples). Thus, it will be critical to understand the central role of humans in (also) complex organizational settings. The new communication tools and platforms can imply new needs of management for social relations and integrate them with connected objects through IoT. It can be imperative not to mix the functions of social media and services with humans as social actors. Together it can transform CPS systems in an "uncanny valley" already described by Mori several decades ago [43].
To conclude, social interaction will become more complex with SNS and CPS [68]. Operator 4.0 will remain a human and not just a "thing" connected with other cyberphysical devices.

Human-inspired social relationships-based sociability model: from social integration to system integration
This third approach consists in a transposition of human-inspired social relationships into a technical (e.g., SIoT) or socio-technical system (associating objects and humans). Some years before, the advent of Industry 4.0, [44] noted a certain lack of consideration for human factors in the field of CPS, and developments were being focused on networked & next-generation embedded systems [2,18,66]. Therefore, he proposed the concept of 'Cyber Physical Social System' (CPSS) as a "tightly conjoined, coordinated, and integrated with human and social characteristics" development of CPS.
CPSS being supported by the addition of physiological, psychological, social, and mental spaces to those of cyber and physical spaces [45,46,63] (Fig.10). Written as the Word from the Editor for the first issues of the CPSS department of IEEE Intelligent Systems journal, this first approach necessarily stays conceptual. Yet, it has quickly be followed by much more concrete works. Figure 10. From Popper's three worlds to cyber-physical social systems. [44] We can notably cite the 'Social Internet of Things' (SIoT), developed by [47] (Fig. 11). Equally based on the identification of the need to structure data into the growing Internet, the goal of this development differs from [48], for it does not focuses on the reuse of existing SNS structures, but rather on the development of a new architecture that would be "a social network of intelligent objects bounded by social relationships" [49]. This is based on 5 main social relationships inspired by human systems, such as those developed by [50]. Figure 11 -Architecture for SIoT: client side (left) and server side (right) [51] According to Fiske [50], human societies are regulated by four elementary forms of sociability, namely: Communal Sharing (CS), Authority Ranking (AR), Equality Matching (EM), and Market Pricing (MP). This work represents a first attempt to create a typology of social relations, which Atzori and colleagues used as a basis to develop their own Preprints (www.preprints.org) | NOT PEER-REVIEWED | Posted: 26 July 2021 doi:10.20944/preprints202107.0557.v1 typology [51]. They defined the following five inter-object relationships: Parental Object Relationship (POR), Ownership Object Relationship (OOR), Co-Working Object Relationship (C-WOR), Social Object relationship (SOR) and Co-Location Object Relationship (C-LOR).
Simultaneously, Atzori and colleagues [47,51] have developed a support architecture for object-object interactions and the discovery of services and resources within a network of connected objects. Social relationships are established and exploited among objects, but not between their human beneficiaries.
Contrasting with previous social approaches, this one relies on human inspired social mechanisms to improve the integration of purely technological systems. However, the relationships expressed in SIoT pave the way for the realization of a paradigm evoked earlier: the 'Cyber-Physical Society'. It encompasses the definition of Society 5.0 already referred above. Valette et al. [69] have proposed a transposition of those relationships to a Social Holonic Manufacturing control in CPS based factories It was defined by Shi and Zhuge (Fig. 12) as a 'Cyber-Physical Socio-Ecosystem' (CPSE) where natural physical space, social space, mental space and cyberspace interact and co-evolve with each other [46]. CPSE deals with the relationships between individuals in a cyber-physical environment and cyber-physical social system. If a robot, or other machine with AI can be equipped with general information about social behavior, than it should be able to detect situations as appropriate in which certain classes of social behaviors and apply them. That is specially the case, where it is needed to include information about possible human operator actions in the programming phase.
With more developed devices, such capacity can also feature the intuition capacity in the interaction with humans [33]. In such case, machines can have an autonomous "reasoning'" about how best to achieve its goals in a given social context and should have the ability to express itself in ways that will help it complete tasks in a wide range of social situations. The expression must not be verbal but can be in written forms to become understood by the human operator.
The frames of goal achievement must be settled in work environments. In other words, the "higher the capacity is for 'autonomous reasoning', the higher must be the intuition for humans to interact with robots" [33]. This means the cyber-physical space must be designed and programmed according to the social system, for example, the one presented in a model of work organization [19,26,52,61]. In such way, we can talk about a potential symbiosis between the physical space (machinery, tools) and social space (mental references, identities, communication language, organizational roles, decision-Preprints (www.preprints.org) | NOT PEER-REVIEWED | Posted: 26 July 2021 doi:10.20944/preprints202107.0557.v1 making) referred also as Cyber-Physical Society or Socio-Ecosystem, or even Social System (CPSS), as mentioned above. This logic is also found in the work of Pintus and colleagues [53]. These authors define the 'Humanized Internet of Things' (HIoT) as a classic Machine-Machine oriented IoT coupled with SIoT and the 'Internet of People' (IoP) [53]. In manufacturing environment, the goal is to propose a social factory supporting human workers under any condition and engaging him to contribute to knowledge creation. In such a system, human, machine, and software agents would be considered as equal and provided "just in time" and "just in quality" with necessary information [54,56,57,58,62].
It is easy to perceive, behind this assemblage of paradigms, a larger vision of a sociotechnical system of agents, artefacts, and human beings, organized governed by a set of social relations. Nevertheless, there is still a lot of work to be done before reaching a better acceptability of these systems, to reduce their complexity while guaranteeing their agility when facing changes of environment and to allow a better integration of the human being either as an individual with its variabilities and as a part of a collective society through the concept of social inclusion [55,64].

Conclusion and Prospects
Automation has been, significantly, increasing in most production sectors, and the question remains, whether there remains space for human autonomy and creativity at working level. Does automation increase the dependency of the workers within these new systems? What are the qualitative changes of work on shop floor level with regard to speed, expectations, demands and complexity of work? How are workers integrated into the configuration of these working arrangements? Why the ongoing process of automation is still implicitly part of development?
Today's new technological advances, associated with CPS and IoT, are paving the way of the new generation of automated production systems with promising objectives of efficiency, agility, and adaptation to user needs. New levels of automation will be accomplished based on these systems with better balance between human and machine works. This is a promise of industry 4.0.
The purpose of this study was to explore how human/social dimension was considering in CPS and IoT based production System. The focus was done on the sociability models to ease the human social/system integration into automated production systems. The first model is dealing with the classical Human-System interaction interfaces where many works are done to propose new interactive interfaces or embedded sensing systems. The second model is aiming to take advantage of the form of existing Social Network Services (as social websites like Facebook, Twitter …) that offer a variety of features facilitating the socialization based on internet. The last model is relating to the design of industrial system as a society linking smart connected objects through a typology of social relations and paving the way to a good human system integration.
Certainly, all these models, described above have as its aim to increase productivity. At the same time the systems are increasing the possibilities for intuitive interaction between humans and machines, and to contribute to ease the working conditions for operator in complex environments. The design of these elements may be a need for most automated environments that apply the concept of industry 4.0 and articulate most CPS in production systems.
When reflecting on such automatization processes is necessary to mention the economic, political, and ethical challenges. There are not only technical ones. The economic challenges have implications on society, and the political and ethical have direct influence on the legal framework. Taking these issues in consideration, number of scientific questions remain open to develop and explore future perspectives for a conscious and human cyber-physical society to reinforces the role and the contribution of industry to society leading for what today we name Industry 5.0 Funding: This research was partially funded by Portuguese national funds through FCT -Foundation for Science and Technology, I.P., within the scope of the project «UIDB/04647/2020» of CICS.NOVA -Interdisciplinary Centre of Social Sciences. It has also the support from the German project Kopernikus -SynErgie II, supported by Federal Ministerium for Education and Research (BMBF).


extfieldsofstudy:
Medicine
Computer Science

text:
UNOC: Understanding Occlusion for Embodied Presence in Virtual Reality

Tracking body and hand motions in the 3D space is essential for social and self-presence in augmented and virtual environments. Unlike the popular 3D pose estimation setting, the problem is often formulated as inside-out tracking based on embodied perception (e.g., egocentric cameras, handheld sensors). In this paper, we propose a new data-driven framework for inside-out body tracking, targeting challenges of omnipresent occlusions in optimization-based methods (e.g., inverse kinematics solvers). We first collect a large-scale motion capture dataset with both body and finger motions using optical markers and inertial sensors. This dataset focuses on social scenarios and captures ground truth poses under self-occlusions and body-hand interactions. We then simulate the occlusion patterns in head-mounted camera views on the captured ground truth using a ray casting algorithm and learn a deep neural network to infer the occluded body parts. In the experiments, we show that our method is able to generate high-fidelity embodied poses by applying the proposed method on the task of real-time inside-out body tracking, finger motion synthesis, and 3-point inverse kinematics.

INTRODUCTION
Thanks to fast wireless Internet and ubiquitous smartphones, an everincreasing fraction of communication takes place virtually. This is even more true during social distancing measures in a pandemic, which have made video calling the de facto standard for everything from corporate meetings to Ph.D defenses. However, among other limitations, video calling lacks the sense of being physically present in the same space. The use of avatars in virtual reality ("embodied VR") is one promising direction which simulates co-presence in a virtual 3D space through spatial audio and body language [38].
Because complicated capture setups requiring outside-in cameras are impractical, the industry is converging on self-contained headsets where cameras mounted on the headset ("inside-out") can be used for both headset and controller tracking. While recent progress has been made using these cameras to track hands [1,26,35], tracking the full body from the same cameras remains a challenge [3,33,43]. Headsetmounted cameras suffer from both occlusion and a limited field of view. When standing, the upper body may occlude the lower body; when seated, the knees may occlude the feet. Users may interlace their hands behind their heads or reach up to scratch their cheek.
To track these challenging cases where visual evidence is lacking, we need strong priors to generate plausible motion from sparse data. Building these priors requires high quality motion data. Existing motion capture databases focus on dynamic motions found in gaming or visual effects scenarios: walking, running, acrobatic jumps. Such motions are uncommon in VR meeting settings.
A survey of public domain CSPAN videos (see Fig. 1) shows that meeting attendees display a variety of behaviors seldom captured on the traditional motion capture stage. They may slouch, cross their arms, rest a cheek against a palm, scratch an ear or press their hands between their knees. These motions have several characteristics in common. First, they involve a combination of the hand and body: capturing only the joints above the wrist will mean missing out on important context. Second, self-contact is an important part of the motion: the hand touches a cheek, or the hands are tucked inside armpits or pockets. Third, much of the motion is occluded and potentially out of frustum for a typical headset camera configuration, making it particularly challenging for inside-out body trackers.
Capturing these motions is even difficult using traditional optical motion capture setups. Due to occlusions, markers are constantly disappearing, and entire limbs may be mis-predicted under a lack of enough markers. We address this challenge in several ways. First, we use a non-line-of-sight IMU-based glove capture system for the hands. Second, we send all captured data (including RGB videos) to a motion capture studio for manual clean-up. Finally, we clean up the final motion with an inverse kinematics algorithm. To adapt the dataset for inside-out body tracking, we propose a simulation algorithm to generate occlusion labels about what headset camera views can observe.
In addition to creating a high quality dataset of meeting motions, we propose a method to predict occluded or out-of-view limbs for insideout body tracking. Specifically, we apply temporal neural networks (i.e., RNN, LSTM [13], GRU [8], TCN [29]) to predict ground truth locations from poses observed in a time window. Our method is designed to be a differentiable post-process for inside-out body trackers, useful as a pose prior that regularizes the output motion.
Evaluating our method on both existing public datasets and our newly collected motion, we show that we can generate plausible embodied poses for both the body and the hands with substantial missing information. We also conduct detailed comparisons of the proposed model applied to two other popular tasks: 3-point inverse kinematics for VR, and finger motion synthesis, which shows that our method overcomes the problem of missing or occluded body joints during realistic interactions between the hands and the body. Our experiments indicate that our data fills a gap in the existing collection of motion capture datasets and enables future research in learning embodiment for social presence in AR and VR.
Contributions With the goal of predicting the pose of missing body joints, we make two major contributions: 1. We collect a large-scale motion capture dataset (UNOC) for inside-out body tracking. The dataset focuses on difficult-totrack motions with hand-body interactions that are common in meetings and other social interactions.
2. We propose a new deep neural network to predict the missing or occluded body parts and demonstrate that our model can generate plausible embodied poses in three applications: inside-out body tracking, finger motion synthesis, and 3-point inverse kinematics. Notice the frequency of self-contact in the CSPAN data including clasped hands, crossed arms, and hand-face contact. In our study, we collect poses "inspired" by the CSPAN data, resulting in a dataset of challenging cases that can be used to train and test machine learning models.

RELATED WORK
Body motion capture datasets Many body motion capture datasets are now available for research. HumanEva [36] and HDM05 [27] contain simple human motions such as running, dancing and sports activities. The CMU Motion Capture Dataset [12] provides a larger variety of human motions including two-person interactions. Human3.6M [15] includes more remote communication scenarios, such as face-to-face discussion and talking on the phone. However, communication-focused poses are limited and all these dataset lack finger poses, which are important for VR presence [42]. In contrast, our dataset focuses on contact-rich scenarios and provides both hand and body pose.
Body and hand motion capture datasets The paucity of motion data capturing both body and hands is best illustrated by AMASS [22], which aggregates research motion datasets. Out of the 15 included datasets spanning 40 hours, only the 8-minute-long TCD Hands [14] provides the hand pose. TCD Hands focuses on hand-object manipulation, pointing and signing using the hands, but contains little body-hand contact. Recently, Taheri et al. [40] released GRAB, a high-quality fullbody optical motion capture dataset recording how humans hold different objects. The resulting motion is designed to maximize the quality of the hand grasping and thus largely avoids the body self-occlusions seen in Fig. 1. Lee et al. [21] released a dataset incorporating body and hand tracking, TWH16.2M. This dataset is a large-scale collection of unscripted conversations between multiple, simultaneously tracked participants, aiming to capture natural gestures happening during conversations. However, the use of markers on the hands means that they cannot be captured when occluded by e.g. crossing the arms, and the resulting motion contains very few of the contact-rich motions that we focus on (see Table 1 for a comparison). We capture a mix of scripted and unscripted motions with markerless gloves to achieve a diverse set of contact-rich poses as well as natural motion during conversations.
Occlusion-aware and temporal body tracking Explicitly modeling occlusion between body parts [31,37] and people [11] has been shown to improve the accuracy of human pose estimation. Various pose priors have also been used to address occlusions, including Mocapguided priors [34], kinematic body models [19], adversarial poses discriminators [5] and adversarial generation of occlusion masks during data augmentation [30]. Temporal information has also been shown to be effective at filling in missing information, and has been particularly successful at predicting depth from 2D joints [2,9,29,32]. VIBE [20] combines adversarial learning and temporal modeling by using a discriminator trained on AMASS [22] to distinguish between real human motions and the output of temporal pose and shape regression networks.
Cheng et al. [6,7] combine both explicit occlusion modeling and temporal networks to filter out occluded 2D joint predictions before lifting them into 3D with temporal convolution networks. We also propose a method of using explicit occlusion modeling and temporal neural networks, but with an application to inside-out body tracking. Our method can serve as an additional pose prior to existing 3D human pose estimation techniques.
Egocentric body tracking estimates body motion from egocentric cameras, which is both highly relevant for embodied presence and also presents highly challenging occlusion. With the advancement of AR/VR technologies, inside-out body tracking using all-in-one headsets has received increasing attention in recent years. One line of work aims to learn plausible body pose without requirement of camera coverage of user's body in egocentric view. Yuan et al. [45] propose to use imitationlearning to learn a video-conditioned control policy and demonstrate that the framework can predict plausible body pose without observing camera wearer's body. Jiang et al. [17] leverage the static scene vs dynamic scene classifier, the dynamic motion signature of the surrounding scene as well as pose coupling in longer time span to form a global optimization problem to infer the pose sequence. Both frameworks assume invisible pose and rely on the assumption of correlation of the user's body motion with dynamics of the surrounding scene, and have been demonstrated only on simple body motions. When body parts are at least partially visible from the cameras, 3D body pose estimation can be applied. Xu et al. [43] use a cap-mounted fish eye camera to capture the body and predict body joint heatmaps and depth, followed by back-projection into 3D camera space to obtain 3D body pose. Tome et al. [41] also predict body joint heatmaps but adapt an encoder with a dual-branch decoder to transform joint heatmaps into 3D pose while accounting for uncertainty of 2D keypoint locations. We consider our work to be orthogonal to and complement these vision-based methods by 1) providing hand and body motion datasets for synthesizing hard-to-capture data for challenging but critical body poses; and 2) as a motion prior that can be incorporated into these methods.
Finger synthesis and 3-point body tracking We evaluate our method on two additional missing data problems: Finger synthesis predicts finger motion from body motion and has been addressed with database driven methods [18,25] and deep learning [21]. 3-point body tracking addresses the common case of VR headsets that track only the head and the two controllers held by the hands [28]. We show that both our proposed dataset and motion prior are effective for these tasks.

PROPOSED DATASET
Dataset overview With a focus on occlusion and contact during social interactions, we built a hand-body mocap pipeline that is robust to occlusion and recorded 220 minutes of motion with 13 adult participants (10 males and 3 females). The dataset comprises 1.6M frames of cleaned data (plus 200K frames for T-poses and other calibration data) at 120 frames per second with 211 recorded sequences (besides 26 for calibration). It captures frequent occlusion and heavy contact between hands, arms, head and other body parts during social interaction, sitting or standing, with or without a table. We measure the prevalence of body-body part contacts in our data by slightly increasing the size of the primitives used for occlusion simulation (see 4.1) and detecting interpenetrations between non-adjacent bones. Compared to MIXAMO [44] and TWH16.2M [21], UNOC has 2x and 10x more body-body part contacts. The hands interact with each other and with the body in 59% of the frames, compared to 24% and 4%. Detailed statistics and comparisons are shown in Table 1.

Motion capture pipeline
To capture occlusion-and contact-heavy motion, we combine traditional optical motion capture (for the body) with an IMU-based system (for the hands). As shown in Fig. 2, our optical tracking system uses 23 Optitrack Prime 22 cameras to capture 57 retro-reflective markers placed around the subject's body using the Optitrack Biomech marker configuration. The tracked markers are manually cleaned and verified for all sequences to prevent mislabeling due to occlusion. The used NANSENSE BioMed gloves utilize 16 IMU sensors for each hand to track the orientation of palm and finger segments. The tracked hand poses are robust to severe or complete occlusion, with acceptable compromises regarding latency and accuracy, and the lack of a calibrated global transform with optical tracking.
We combine the solved body and hand motions by using the finger joint transforms from the glove-tracked hand, while using the hand orientation from the optical system. We found that the optical-tracked hand orientation has a different yet consistent offset for each participant due to the placement of sparse markers on the wrists and hands. We remove the differences by an additional calibration sequence in which we carefully place retro-reflective markers on the fingertips and compute the offset by rotating and aligning the optical-tracked hands with the auxiliary markers.

Data capture protocol
We recruited 13 adult participants (10 male, 3 female) comfortable wearing a motion capture suit to go through sessions that typically took 1.5 hours each. At the beginning of each session, the participant changed into a velcro-covered mocap suit; a researcher attached 57 retro-reflective optical markers based on the Optitrack Biomech configuration; the participant then donned a pair of NANSENSE BioMed gloves. We first captured calibration sequences for hands and body that consisted of exercising each joint independently. Next, we collected a few "warm-up" motions: idling, walking, crouching, jumping, talking on a phone, giving a thumbs up. Afterward, we focus on the collisionand contact-heavy "social" poses that are frequent in meetings. To capture a mix of sitting and standing poses, we used a stool and a table that could be rolled in and out of the session. We brought the table in mid-way through the capture session and adjusted its height to be comfortable for the subject. The table used a single leg to minimize occlusion and was heavily taped to reduce reflections. Toward the end of the session, the participant would take part in a two-to-five-minute unscripted conversation with the researcher, cued by a question; e.g., "What did you do over the weekend?" or "What was the last movie you watched?" We found that videos were an effective way to convey to participants what types of motions to perform. We studied CSPAN public domain videos from the United States Congress. From these, we collected a set of 52 short segments to present to participants. People in these clips perform a wide-ranging set of motions, including: resting their chins on their wrists, scratching their head or their ears, interlacing their fingers, gesturing, and crossing their arms. The full set of videos can be seen in the supplemental material. In an early internal test, we found that asking participants to mimic particular motions resulted in stiff, unnatural movement. Therefore, we instead asked participants to use the clips only as inspiration. Subjects used the videos in different ways, depending on their preference: some would watch the video as they performed; some would watch the video once and then come up with their own poses; and some preferred asking the researcher to demonstrate the poses. In all cases, we emphasized that participants try to be as relaxed and natural as possible and take as much time as necessary to complete the capture.

PARTIAL BODY POSE PREDICTION
In this section, we discuss our method for predicting the full body pose when some joints are occluded or out of the camera frustum.

Occlusion simulation
To evaluate our network on its capability to reconstruct untracked joints, we annotate the joint locations in the ground truth data with a label specifying if a given joint is occluded or outside the camera frustum. To simulate the multiple inside-out tracking cameras of headsets like the Oculus Quest or Microsoft Hololens, we place a virtual camera four centimeters in front of the nose. The camera uses a circular fieldof-view of 200 • , and is tilted downwards by 15 • . Joints are tagged "out-of-view" if they are outside this cone. In addition, joints closer than 12cm to the camera center are considered untracked because they would typically be out of focus and lack visual context.
To simulate occlusion efficiently, we use a simplified geometric model where each bone segment is represented by a capsule or bounding box [10]. We test visibility by tracing rays from the camera center and check whether they intersect any primitive before reaching the skeleton. A given body joint is considered visible if the joint itself is visible as well as at least one primitive attached to the joint (see Fig. 4). In our experiments, this heuristic works well and requires less fine tuning than using an image-based occlusion simulation (i.e. counting visible pixels per body part, or training a neural network on our inside-out rendered views).
In this manner, we test for the visibility of the shoulders, elbows, hips, and knees. We treat the feet specially, as the ankle joint itself is almost always occluded; instead, all the joints in the feet are considered visible if any (including the toes) is visible from the camera. A similar procedure applies to the hand: while fingers frequently self-occlude, evidence from the vision-based tracking literature suggests it is possible to predict a reasonable hand pose even when some digits are hidden [46]. We therefore mark the entire hand as visible provided at least 65% of its primitives are visible.

Embodied pose prediction
Given these occlusion annotations, we can define the partial pose prediction problem. Given the 3D locations of all the joints that are tracked, we will predict the 3D locations of all the body joints. While we focus here on 3D joint locations, these can be post-processed into  a fully posed human body model using an inverse kinematics (IK) solver [28,39].
It is often intractable to predict missing joints from single-frame inputs. Motivated by [4,29], we incorporate the history and tackle the problem through a temporal model. While extrapolation based on constant position velocity is a reasonable baseline [24], the best extrapolation is likely situation-dependent: a palm supporting a cheek should be pinned relative to the head pose, while a gesturing hand might finish resting on a hip. Therefore, we train models that learn to extrapolate from partially observed present and historical joint locations.
Proposed architecture Our network takes the last 27 tracked frames (at 30fps, this results in a receptive field of 0.9 seconds) as input (i.e., time window size T=27). For occluded or out-of-view joints, we repeat the last known position. Our network backbone uses Gated Recurrent Units (GRU) [8] with a 512-dimensional hidden state size, which formulates the sequential data into a hidden Markov Chain model and describes relations among neighboring nodes with sharing weights.
For a node t representing the t-th frame in the sequence, we use t − 1 and t + 1 to denote the previous frame and the next frame, respectively. Given an input feature encoding x t for node t, the output y t is determined jointly by the hidden states (i.e., linear weights W and bias term b), update gate z and reset gate r: whereŷ denotes the candidate activation, denotes the Hadamard product, σ denotes the sigmoid function and φ denotes the hyperbolic tangent function. The model takes the (partially occluded) 3D joint locations as input, and the result of the GRU is fed through two branches of fully connected layers to predict 3D positions and occlusion labels, respectively (see Fig. 3).
In an egocentric setting, the 3D body joint positions are given as offsets from the head location, while the fingers are given as wristlocal joint positions, removing inherited inaccuracies from the wrist position. Our experiments show that this helps the network to predict small nuances in gestures and finger poses, without being penalized for the global positional error (see Table 2). We ask our network to predict whether a given joint is occluded, rather than giving the occlusion mask as input to encourage it to model occlusion. This has similar accuracy to providing the occlusion mask as input, but the transitions between visible and occluded joints in the prediction are noticeably smoother. The percentage of frames exhibiting strong acceleration (> 9m/s 2 ) is reduced from 6.1% to 3.5% by predicting the occlusion mask (compared to 0.2% in the ground truth). In the final architecture, the output of the GRU is concatenated with the input through skip connections and branched out to two MLPs with the number of hidden neurons {512, 512} for two tasks: joint location prediction and occlusion classification.
Losses and training To train the network, we use a weighted sum of multiple losses. The total loss function is defined as L = L pos + L kin + L occ , (2) where L pos denotes the global position loss, L kin the kinematic loss, and L occ the occlusion mask loss. For the global position loss and the kinematic loss, we measure them by the Euclidean distance D(·, ·).
L pos measures the distances between the ground truth joint position p i and the predicted joint position p i in the head/wrist-local frame as described above. We weigh finger joint errors higher than body joint, and occluded joints higher than visible, i.e., L kin penalizes violations of kinematic limits of the skeleton, helping to ensure that the bones {b i, j = p i − p j } are valid in the predicted pose. The first term measures the differences on the parent-local joint positions b i and the second term measures errors in bone lengths, that is, where ch(i) denotes the children joints of joint i in the kinematics tree. L occ penalizes the misclassification of the predicted occlusion labels against ground-truth, i.e., where H(, ) denotes the cross entropy for the binary classification.
In our experiments, we use a weight of 10 for the head-local body joint position loss L pos , and 100 for the wrist-local finger joint position loss, both weights are increased by 20% if the joint is occluded. We weigh the parent-local position loss by 3, which is also increased by 20% for occluded joints. We weigh the bone length errors of body joints by 7 and fingers by 10 for L kin . The weight for the occlusion mask loss L occ is set to 0.025. We implement our method in PyTorch and train it using the Adam optimizer with a learning rate starting from 1e-3. The models are trained over 5 epochs, with a batch size schedule of {256, 512, 1024, 1024, 1024}.
Post-processing 3D full-body avatars are typically animated using forward kinematics, where the pose is encoded using local joint angles for each bone relative to its parent. Because the output of our networks are 3D world-space joint positions, computing the pose requires solving for the body pose using inverse kinematics (IK). Our IK solver computes the body joint angles that respect the hard bone-length constraints while minimizing the world-space positions' squared error by the Gauss-Newton algorithm, initialized from a rest pose for the first frame, and the previous frame subsequently. We use fixed-offset auxiliary joints at the end of the kinematic chain (e.g. finger tips) to disambiguate joint orientations. When joints become visible after being occluded for a while, the neural network snaps the joints to their new location which results in fast, unrealistic motions. We experimented with temporally consistent networks, however these networks traded accuracy for smoothness. Using a simple momentum-based algebraic smoothing instead, we achieve very smooth motion without sacrificing accuracy (see Section 5.1).

EVALUATION
We evaluate our proposed method and the UNOC dataset on three applications. First, we compare inside-out body tracking using both the occlusion simulation and occlusion prediction method described in Section 4. We show that our proposed method generates smooth and plausible motion for this application, and that our dataset is critical for covering new motions for embodied presence. Because inside-out body tracking is a nascent area where camera placement and datasets have not been standardized, we also show that our method is effective on two additional applications: 3-point upper body tracking and finger synthesis. In these tasks, the missing information does not vary across frames, so we eliminate the occlusion mask prediction branch of our method. For all applications, we use an 80%/20% train/test split and ensure that the test set includes two participants not seen during training. We use root mean square per-joint position error (RMSJPE) and mean per-joint position error (MPJPE) as the evaluation metric. All units are in centimeters.

Inside-out body tracking
We first evaluate our proposed body prediction method (see Section 4.2) on three datasets: our own UNOC dataset, Mixamo [44] and TWH16.2M [21]. For each dataset, we first simulate the output of an inside-out body tracking system using the occlusion simulation step described in Section 4.1 and then train a model to evaluate the tracking accuracy. To the best of our knowledge, there is no other published method that is applicable to this complex missing data problem. We experimented with applying IK solutions and smoothing for the baseline (see the user study description), which increased the overall error. To this end, we set up a baseline, which copies the last tracked position for occluded joints while using the ground truth for tracked joints. As shown in Table 2 (first two columns) and Fig. 5, our method is able to significantly reduce the joint positional error over the baseline model. For occluded body joints, our methods approximately halves the error; for occluded fingers we reduce it by nearly one centimeter. In Table 5 (left), we show how our model trained on one dataset performs when predicting body poses on other datasets. It can be observed that our dataset is unique and the set of motions that we use in our animations are not covered by other datasets.
Ablation studies We conduct an ablation study of different model architectures and representations, as shown in Table 2 and 3. We first try replacing GRU with a few other popular sequential neural networks (RNN, LSTM [13], TCN [29]). For a fair comparison, we use the same hidden state size as our proposed architecture for RNN and LSTM and we use the causal convolution version for TCN to be consistent with our online inference setting. From the results, it can be observed that GRU performs best in our experiment setting compared to other temporal neural networks. We further evaluate several different variants of the proposed network architecture, i.e., with/without skip connections,     Table 4: Ablation study of losses for inside-out body tracking on UNOC dataset. We use RMSJPE/MPJPE to measure tracking errors of occluded/body/all joints in cm. Parent local error can be understood as bone length error. different coordinate system representations, different time window sizes and different hidden state sizes. From the results, we find that 1) adding skip connections helps preserve visible joint positions and generates smoother results, 2) wrist-local finger coordinates improve accuracy over head-local (i.e., our general body representation) and 3) the current hyper-parameter configuration obtains fairly good performance. Furthermore, we conduct an ablation study of different loss combinations, as well as when removing the occlusion prediction branch, see Table 4. The results show that predicting the occlusion masks improves the joint accuracy of occluded joints. Removing the parent local position error loss, the bone length loss, or both at once, greatly increases the bone length errors compared to using both losses together.
User study To assess the practical value of our approach, we conducted a user study. Due to COVID-19, we opted for a video comparison study rather than an actual VR study. From our test data set, we randomly selected eight sequences with varying motions and levels of occlusion, including: leaning on a table, walking, scratching the head, touching the face, finger gestures, drinking from a glass, running, and crossing arms. For each of these animations, we created renderings with our standard avatar using three different methods: the fully captured ground-truth (GT), using the last-known position for each joint (BASELINE), and using the predictions from our trained network (OURS). In both of the test conditions (BASELINE and OURS), rendering the body mesh requires converting from 3D positions to joint angles, so we use the IK solver described in Section 4.2 (configured identically for both cases) and apply the same momentum-based postsolve smoothing. After the smoothing and the IK solver stage, the root mean square joint position error in OURS decreases by 0.04cm for occluded body joints, while the overall error increases by 1.5cm compared to the network output positions. The root mean square joint position error of occluded body joints in the BASELINE was reduced by 3.5cm, while the overall error was increased by 2.4cm using the smoothed, IK solved positions.
In our study, we showed side-by-side views of all pairs of renderings to the participants, following a "similarity judgment" design [23]. For each pair, the participants were asked to choose the more natural result ("Which animation is more natural?"). They could choose either video or indicate that they were equally natural. To avoid biases and learning effects, we randomized the order of pairs as well as the position of the videos while ensuring that for each participant the same number of techniques were shown on the left and right. To evaluate the results, we attribute the choice of one technique with +1 and the other with −1. Averaging over all eight scenes results in a quality score for each pair of techniques. Using a t-test, we determine the probability of the drawn sample to come from a zero-mean distribution-zero-mean would indicate both techniques to be of equal quality.
We recruited 39 participants with medium to high experience with VR from a local university. Each pair-wise comparison video was combined into a single clip to allow for synchronous playback. The participants were told that they could replay and navigate in the video as they wished.
A detailed analysis of the individual animations further showed that our method achieves significantly better results than the baseline in seven out of eight cases. Only one animation with limited occlusions, where the avatar is standing straight and showing some hand signs (peace, thumbs up, etc.) could not achieve a significant improvement. This might be due to a reduced accuracy in the hand pose: the predicted hand looks slightly closed compared to the tracked pose. Furthermore, the ground truth achieved significantly better results than our model in only five out of eight cases. Two such cases are walking and running animations. In the six remaining sequences, containing body-body part contact and hand-object interaction, we achieve mean scores between -0.18 up to 0.61, three of which see a significant preference for the ground truth.
The results of the study show that a learned prior using our approach clearly creates more realistic animations compared to the baseline which uses only an IK solver and a smoothness term on the last tracked positions. Post-study interviews revealed that the baseline animations were perceived as partially unnatural and jerky. Some poses chosen by the baseline clearly resulted in "uncomfortable poses humans would not choose freely." Furthermore, the animations generated by our solution were described as "completely plausible" and "very natural". However, a direct comparison to ground truth animations still shows a quality gap. This is not surprising though, as even with good priors, predicting hidden motions will not produce sufficient detail unless random animations are hallucinated even when the tracked body pose is not changing (e.g. fingers typing on a keyboard). Participants tellingly described the differences as "lacking detail" or "lacking depth." Still, overall our approach can clearly help provide realistic and natural animations for hidden joints.

3-point upper body tracking
We additionally evaluate our method on 3-point upper body tracking, which is a common missing information scenario for VR headsets that track only the headset and two controllers [16,28]. We compare our method against the method of Parger et al. [28] on both our UNOC dataset and on Mixamo. Because the occlusion mask no longer varies between frames, we drop the occlusion mask prediction branch from our method. In this scenario, we use the transforms of the head-mounted device and the two controllers, since they are tracked by built-in or external sensors. The orientation is represented as forward and upward vectors of the three joints, and used together with their positions as input for the neural network.
As the inverse kinematics approach [28] does not support sitting, we also include an evaluation restricted to standing poses. As reported in Table 6, our data-driven method obtains consistent improvement against IK solver-based methods. It is worth noting that the proposed method reduces the reconstruction errors by at least 50% for the neck, and approximately 66% for shoulder and elbows. We further evaluate the generalization ability of our method by performing cross-dataset validation (i.e., training with one dataset and testing on the other). Though the two datasets have quite different statistics, we still observe improvements led by the universal knowledge learned from data and the error is reduced by about 50% compared with the IK solver-based method (see Table 6 †). Please refer to the accompanying video for detailed comparisons.

Finger synthesis from body motion
Researchers have proposed various approaches for generating finger motion from body motion, including using database-retrieval methods [18] and audio [21]. However, directly comparing with these approaches is difficult as motion segments, models, code and datasets are seldom available. We slightly adjust our model to solve this task. Specifically, the input for our method is the observed body-only poses (i.e., the positions of all body joints together with the up and forward vectors of the wrists) in a time window of 27 frames, which includes the current frame plus 26 frames of history; 2) the output from our method is the synthesized 32 wrist-local joint positions of the fingers for the current frame. We further include one additional output representing the locator on each finger tip to infer the orientation of the distal bone, resulting in 42 output positions all together.
As can be seen in Fig. 7 and in the accompanying video, our network is able to create highly plausible predictions. This is especially surprising as one cannot assume that finger locations can in general be inferred from general body motions. We assume that the high quality of the results can be explained by the fact that body motions are often related to achieving a specific goal which includes finger and hand motion. For example, folding arms, scratching ones head, or leaning on ones hand. Especially in these hand-body contact situations, our approach achieves highly natural results although no finger information is ever presented during testing. It is worth noting that our model cannot generate finger  Fig. 6: Results of our user study. Values > 0 show a preference of the second technique over the first, i.e. GT over OURS, OURS over BASELINE and GT over BASELINE. Error bars show standard error. * indicates statistical significant difference between the methods according to a t-test (all significant results achieved at least p < .01 except for gestures for BASELINE vs. GT with p < .02).  Table 5: Cross-dataset evaluation for inside-out body tracking (left) and finger synthesis (right). The prediction is measured by RMSJPE/MPJPE for occluded joints and finger poses in cm, respectively. animations when the person keeps the static body pose for a while, which is similar to the inside-out body tracking scenario. For example, some hand-only motions, e.g., typing on a keyboard, or scratching the head cannot be synthesized as the model lacks clear semantics about what type of action/interaction is ongoing. We further show the value of our proposed dataset by cross validating our model across Mixamo, TWH16.2M and our UNOC datasets (see Table 5, right). From the results, we can observe that the model learned on our dataset obtains better generalization abilities than models trained on TWH16.2M and Mixamo datasets. This suggests our dataset covers a unique set of finger motions that has not been captued and covered by other datasets.

CONCLUSION
In this paper, we build a motion-capture dataset with simultaneous body and hand tracking. We demonstrate that our UNOC dataset fills the gap of difficult hand-body interactions and occlusions for the purposes of capturing and predicting rich social interactions. We show one application of UNOC by training networks to achieve accurate prediction of missing joints. We hope our dataset will promote advancement on body tracking and immersive virtual communication. The dataset (solved pose and cleaned marker positions), source code and pre-trained models are available at link removed for review.