text:
Unsharp Mask Guided Filtering

The goal of this paper is guided image filtering, which emphasizes the importance of structure transfer during filtering by means of an additional guidance image. Where classical guided filters transfer structures using hand-designed functions, recent guided filters have been considerably advanced through parametric learning of deep networks. The state-of-the-art leverages deep networks to estimate the two core coefficients of the guided filter. In this work, we posit that simultaneously estimating both coefficients is suboptimal, resulting in halo artifacts and structure inconsistencies. Inspired by unsharp masking, a classical technique for edge enhancement that requires only a single coefficient, we propose a new and simplified formulation of the guided filter. Our formulation enjoys a filtering prior from a low-pass filter and enables explicit structure transfer by estimating a single coefficient. Based on our proposed formulation, we introduce a successive guided filtering network, which provides multiple filtering results from a single network, allowing for a trade-off between accuracy and efficiency. Extensive ablations, comparisons and analysis show the effectiveness and efficiency of our formulation and network, resulting in state-of-the-art results across filtering tasks like upsampling, denoising, and cross-modality filtering. Code is available at \url{https://github.com/shizenglin/Unsharp-Mask-Guided-Filtering}.

I. INTRODUCTION
I MAGE filtering has been widely used to suppress unwanted signals (e.g., noise) while preserving the desired ones (e.g., edges) in image processing tasks like image restoration [1]- [3], boundary detection [4]- [6], texture segmentation [7]- [9], and image detail enhancement [10]- [12]. Standard filters, such as Gaussian filters and box mean filters, swiftly process input imagery but suffer from content-blindness, i.e., they treat noise, texture, and structure identically. To mitigate contentblindness, guided filters [13]- [18], have received a great amount of attention from the community. The key idea of guided filtering is to leverage an additional guidance image as a structure prior and transfer the structure of the guidance image to a target image. By doing so, it strives to preserve salient features, such as edges and corners, while suppressing noise. The goal of this paper is guided image filtering.
Classical guided filtering, e.g., [13], [19]- [21], performs structure-transferring by relying on hand-designed functions. Nonetheless, it is known to suffer from halo artifacts and structure inconsistency problems (see Fig. 1), and it may require a considerable computational cost. In recent years, guided image filtering has advanced by deep convolutional neural networks. Both Li et al. [22] and Hui et al. [23] demonstrate the benefits of learning-based guided filtering over classical guided filtering. These works and their follow-ups, e.g., [24]- [26], directly predict the filtered output by means of feature fusion from the guidance and target images. Yet, The authors are with the Informatics Institute of the University of Amsterdam, Amsterdam, the Netherlands. this implicit way of structure-transferring may fail to transfer the desired edges and may suffer from transferring undesired content to the target image [26], [27].
Pan et al. [27] propose an alternative way to perform deep guided filtering. Rather than directly predicting the filtered image, they leverage a shared deep convolutional neural network to estimate the two coefficients of the original guided filtering formulation [13]. While their approach obtains impressive filtering results in a variety of applications, we observe their network has difficulty disentangling the representations of the two coefficients, resulting in halo artifacts and structure inconsistencies, see Fig. 1. Building on the work of Pan et al. [27], we propose a new guided filtering formulation, which depends on a single coefficient and is therefore more suitable to be solved by a single deep convolutional neural network.
We take inspiration from another classical structuretransferring filter: unsharp masking [28]- [31]. From the original guided filter by He et al. [13] we first derive a simplified guided filtering formulation by eliminating one of its two coefficients. So there is only one coefficient left to be estimated for deciding how to perform edge enhancement, akin to unsharp masking. To arrive at our formulation, we rely on the filtering prior used in unsharp masking and perform guided filtering on the unsharp masks rather than the raw target and guidance images themselves. The proposed formulation enables us to intuitively understand how the guided filter performs edge-preservation and structure-transferring, as there is only one coefficient in the formulation, rather than two in [27]. The coefficient explicitly controls how structures need to be transferred from guidance image to target image and we learn it adaptively by a single network. To that end, we introduce a successive guided filtering network. The network obtains multiple filtering results by training a single network. It allows a trade-off between accuracy and efficiency by choosing different filtering results during inference. This leads to fast convergence and improved performance. Experimental evaluation on seven datasets shows the effectiveness of our proposed filtering formulation and network on multiple applications, such as upsampling, denoising, and cross-modality filtering.
II. BACKGROUND AND RELATED WORK In guided filtering, we are given an image pair (I, G), where the image pair has been properly registered by default. Image I needs to be filtered, e.g., due to the presence of noise or due to low resolution because it is captured by a cheap sensor. Guidance image G contains less noise and is of high resolution with sharp edges, e.g., because it is captured by accurate sensors under good light conditions. The low-quality image I can be enhanced by filtering under the guidance of highquality image G. Such a guided filtering process is defined Fig. 1: Motivation of this paper. We show an example of depth upsampling (16×) using an RGB image as guidance. Both the conventional guided filter by He et al. [13] and the state-of-the-art deep guided filter by Pan et al. [27] explicitly estimate two coefficients, respectively (ā,b) and (f α , f β ). In their current formulation, however, both methods are likely to over-smooth edges (compare edges in blue boxes) and transfer unwanted textures (compare highlighted details in red boxes). Our proposed guided filter, taking inspiration from unsharp masking, only requires learning a single coefficient f a (notation details provided in Sections 2 and 3). As a result, we obtain a more desirable upsampling result, free of undesirable structures and textures from the guidance image. asÎ = F(I, G), where F(·) denotes the filter function and I denotes the filtered output image. Below, we first review the benefits and weaknesses of classical guided filter functions and existing deep guided filter functions. We then present how our proposal can be an improved guided filtering solution by establishing a link to unsharp masking.

A. Classical guided filtering
The guided image filter [13] assumes that the filtered output imageÎ is a linear transform of the guidance image G at a window w k centered at pixel k: where a k and b k are two constants in window w k . Their values can be obtained by solving: Here, is a regularization parameter penalizing large values for a k . The optimal values of a k and b k are computed as: Here,Ḡ k and σ 2 k are the mean and variance of G in w k ,Ī k is the mean of I in w k , and |w| is the number of pixels in w k . For ease of analysis, the guidance image G and filtering target image I are assumed to be the same [13], although the general case remains valid. As a result, we can obtain: Based on this, the regions and edges with variance (σ 2 k ) much larger than are preserved, whereas those with variance (σ 2 k ) much smaller than are smoothed. Hence, takes control of the filtering. However, the value of in the guided image filter [13] is fixed. As such, halos are unavoidable when the filter is forced to smooth some edges [13], [19]- [21]. An edge-aware weighted guided image filter is proposed in [20] to overcome this problem, where varies spatially rather than being fixed: where λ and ε are two small constants and σ 2 k is the variance of G in a 3 × 3 window centered at the pixel k. Γ G measures the importance of a pixel k with respect to the whole guidance image. Kou et al. [19] propose a multi-scale edge-aware weighted guided image filter, in which Γ G is computed by multiplying the variances of multiple windows. By taking the edge direction into consideration, Sun et al. [21] further improve the filter's performance. However, predefined parameters still remain in all these three methods.
Another limitation of classical guided image filters is their assumption that the target image and the guidance image have the same structure. In practice, there are also situations conceivable where an edge appears in one image, but not in the other. To address this issue, recent works [32]- [36] enhance guided image filters by considering the mutual structure information in both the target and the guidance images. These methods typically build on iterative algorithms that minimize global objective functions. The guidance signals are updated at each iteration to enforce the outputs to have similar structure as the target images. These global optimization methods generally use hand-crafted objectives that usually involve fidelity and regularization terms. The fidelity term captures the consistency between the filtering output and the target image. The regularization term, typically modeled using a weighted L2 norm [37], encourages the filtering output to have a similar structure as the guidance image. However, such hand-crafted regularizers may not transfer the desired structures from the guidance image to the filtering output [26], [27]. For this reason we prefer a guided filtering solution based on end-to-end deep representation learning.

B. Deep guided filtering
Li et al. [22], [26] introduce the concept of deep guided image filtering based on an end-to-end trainable network. Two independent convolutional networks f I and f G first process the target image and guidance image separately. Then their outputs from the last layers are concatenated and forwarded to another convolutional network f IG to generate the final output. We define the method as: where ⊕ denotes the channel concatenation operator. Several works follow the same definition but vary in their feature fusion strategies [23]- [25]. AlBahar et al. [24] introduce a bi-directional feature transformer to replace the concatenation operation. Su et al. [25] propose a pixel-adaptive convolution to fuse the outputs of networks f I and f G adaptively on the pixel level. Hui et al. [23] propose to perform multi-scale guided depth upsampling based on spectral decomposition, where low-frequency components of the target images are directly upsampled by bicubic interpolation and the highfrequency components are upsampled by learning two convolutional networks. The high-frequency domain learning leads to an improved performance. Wu et al. [38] alternatively combine convolutional networks and traditional guided image filters. Two independent convolutional networks f I and f G first amend the target image and the guidance image, and then feed their outputs into the traditional guided image filter F IG [13]: Rather than directly predicting the filtered image, Pan et al. [27] leverage deep neural networks to estimate the two coefficients of the original guided filtering formulation [13] based on a spatially-variant linear representation model, leading to impressive filtering results. It is defined as: where f α and f β are two convolutional networks, and denotes element-wise multiplication. To reduce the complexity of learning, in the implementation Pan et al. [27] learn a single network and predict an output with two channels, one channel for f α and another channel for f β . However, we observe that the shared network has difficulty disentangling the representations of the two coefficients, resulting in halo artifacts and structure inconsistencies. Differently, we propose a new guided filtering formulation, which depends on a single coefficient only and is therefore more suitable to be solved by a single deep convolutional neural network. The aforementioned existing deep guided filtering works implicitly perform structure-transferring by learning a joint filtering network, usually resulting in undesired filtering performance [27], [39], [40]. Recent works [39]- [45] take inspiration from coupled dictionary learning [46], and incorporate sparse priors into their deep networks for explicit structuretransferring. Marivani et al. [40]- [43] propose a learned multimodal convolutional sparse coding network with a deep unfolding method for explicitly fusing information from the target and guidance image modalities. Deng et al. propose a deep coupled ISTA network with a multimodal dictionary learning algorithm [45], and a common and unique information splitting network with multi-modal convolutional sparse coding [39], for the sake of explicitly modeling the knowledge from the guidance image modality. Most of these works focus on guided image super-resolution. In this work, we propose an explicit structure-transferring method for general guided filtering problems. In particular, we propose a guided filtering formulation with a single coefficient, and we learn to estimate the coefficient to explicitly decide how to transfer the desirable structures from guidance image to target image. As we will demonstrate, this leads to more desirable filtering results.

C. Unsharp masking
Our formulation is inspired by the classical sharpness enhancement technique of unsharp masking [28]- [31], which can be described by the equation: where an enhanced image is represented byÎ, an original image by I, an unsharp mask by (I − F L (I)) where F L denotes a low-pass filter like Gaussian filters or box mean filters, and an amount coefficient by λ which controls the volume of enhancement achieved at the output. Essentially, guided filtering shares the same function of edges enhancement as unsharp masking by means of the structure-transferring from an additional guidance image. Based on this viewpoint, we derive a simplified guided filtering formulation from the original guided filter [13], with only one coefficient to be estimated, akin to the formulation of unsharp masking in Eq. (10).

III. FILTERING FORMULATION
Here, we outline our new guided filtering formulation, in which only one coefficient needs to be estimated. Compared to estimating two coefficients (a, b) as in the original guided filtering formulation and subsequent deep learning variants, our formulation is more suitable to be solved with one single deep network. We start the derivation of our guided filtering formulation from the classical guided filter [13], summarized in Eq. (1), Eq. (3) and Eq. (4). In Eq. (1),Î is a linear transform of G in a window w k centered at the pixel k. When we apply the linear model to all local windows in the entire image, a pixel i is involved in all the overlapping windows w k that covers i. In this case, the value ofÎ i in Eq. (1) is not identical when it is computed in different windows. So after computing (a k , b k ) for all windows w k in the image, we compute the filtered output imageÎ i by averaging all the possible values ofÎ i with: Similar in spirit to unsharp masking, summarized in Eq. (10), we want to maintain only the coefficient a to control the volume of structure to be transferred from guidance G to the filtered output imageÎ. Thus, we put Eq. (4) into Eq. (11) to eliminate b, and obtain: Next, we rewrite the formulation as: where I i = 1 |w| k∈wiĪ k . SinceḠ k is the output of a mean filter, it's assumed thatḠ k is close to its mean in the window w i . Next we rewrite Eq. (13) as followŝ whereā i = 1 |w| k∈wi a k , and G i = 1 |w| k∈wiḠ k . For convenience, we will omit subscript i in the following.
The formulation in Eq. (14) enables us to intuitively understand how the guided filter performs edge-preservation and structure-transferring. Specifically, the target image I is first smoothed to remove unwanted components like noise/textures, and the smoothing result is denoted by I. However, the smoothing process usually suffers from the loss of sharp edges, leading to a blurred output. To enhance the edges, an unsharp mask (G − G) with fine edges generated from the guidance image G is added to I under the control of the coefficient a, leading to the structure being transferred from the guidance image to the filtered output imageÎ. Finally, we rewrite Eq. (14) to obtain a more general formulation for deep guided filtering: where I m = I − F L (I) and G m = G − F L (G) denote the unsharp masks of the target image and the guidance image, which contain the structures of the guidance and the target images. F L denotes a linear shift-invariant low-pass filter like the Gaussian filter or the box mean filter. f a denotes the amount function, which controls the volume of structure to be transferred from the guidance image to the filtered output image. Next, we will elaborate on this function. Amount function f a . The output of f a is the volume of the structure to be transferred from the guidance image to the filtered output image. Thus, the input of f a should involve the structure of both the target and the guidance image, which together determine the output, i.e., f a (I m , G m ). Ideally, f a should determine the structure-transferring in a pixel-adaptive fashion. It can be a manually designed function as the function a of the guided filter in Eq. (3). It also can be estimated by learning a deep neural network. Compared to hand-crafted functions, learnable functions are more flexible and allow for a better generalization to various image conditions. Successive filtering. Successive operations of the same filter generally result in a more desirable output, thus we develop a successive guided filtering based on our formulation in Eq. (15). Instead of directly iterating the filtering outputÎ, we iterate the outputs of f a as the function decides the effect of filtering. Let f a be a composition of a set of basic functions {f in which • denotes the function composition operation, such as ). With f a we obtain a successive filtering formulation from Eq. (15), In the next section we will detail how to implement our filters with deep convolutional neural networks.

IV. FILTERING NETWORK
There is a function f a in our formulation, which governs a guided filtering coefficient. We propose to solve this function with a single convolutional neural network.
Network for amount function f a . The function f a decides how to transfer the structure of the guidance image to the filtered output image. There are two inputs G m and I m for this function. Two options are available for the network architecture. Like [23], [26], we can separately process these two inputs with two different sub-networks at first, and then fuse their outputs with another sub-network. Alternatively, we can concatenate these two inputs together and forward the concatenation into a single network, similar to the framework of [27]. Empirically, we find that the second option is easily implemented and achieves a desirable filtering accuracy and efficiency. Thus, we design the network of f a with the second option in this work.
In our approach the unsharp masks, rather than the raw images themselves, are used as the inputs of the network. The unsharp mask is more sparse than the raw image itself, since most regions in the unsharp mask are close to zero. The learning of spatially-sparse data usually requires a large receptive field. The dilated convolution is a popular technique to enlarge the receptive fields in convolutional networks [47], [48]. We design our network by cascading four dilated convolutional layers with increasing dilation factors, where all the dilated convolutional layers have the same channel number of 24 and the same kernel size of 3 × 3. Their dilation factors are set to {1, 2, 4, 8}. Leaky ReLU with a negative slope of 0.1 is used as an activation function after each dilated convolutional layer. Finally, a 1 × 1 convolution layer generates the target output for f a .
Network for successive filtering. To develop a network for the successive filtering formulation in Eq. (17), we consider the network of the basic functions {f (l) a } L l=1 as a convolutional block, as shown in Fig. 2 (a). Then stacking this block multiple times results in a deep network for f a . There are two outputs in the block. By concatenating its input and its feature maps from the last layer results in the first output. The concatenation output allows feeding the previous multi-level outputs to the following blocks, leading to improved performance. We develop the second output by using a convolutional layer on top of the last layer of the block, for the sake of balancing accuracy and efficiency.
Stacking more blocks results in higher accuracy at the expense of an increased computational complexity. To allow users to choose between accuracy and computational complexity, we generate filtering outputs from each block. If users want to obtain filtering results fast, the filtering results from the first blocks can be used. When the accuracy is more important, the filtering results from the later blocks can be used. In short, we obtain multiple filtering results while training one single network. The overall network architecture is visualized in Fig. 2 (b).
Optimization. During training, we are given N samples {(I n , G n , Z n )} N n=1 , with I n ∈ I the target image, G n ∈ G the guidance image and Z n ∈ Z the task-dependent groundtruth output image. Our goal is to learn the parameters θ of the network for f a . Two types of loss, L 1 loss and L 2 loss, have been widely used in deep guided filtering works. Early works, like [22], [23], have adopted a L 2 loss, while recent works, like [24], [27], prefer a L 1 loss because it is less sensitive to outliers and leads to less blurry results compared to a L 2 loss [49], [50]. Following these recent works, we minimize the difference between filtered output imageÎ and its ground-truth Z using the L 1 loss, which is defined as: V. EXPERIMENTS In this section, we provide extensive experimental evaluations. Section V-A introduces the experimental setup. Sections V-B, V-C, V-D, and V-E emphasize ablations, comparisons and analysis. Sections V-F, V-G, and V-H show further qualitative and quantitative results, and state-of-the-art comparisons on various applications, including upsampling, denoising, and cross-modality filtering.

A. Experimental setup
Image upsampling datasets. We perform upsampling experiments on NYU Depth V2 [51], and Sintel optical flow 24  [52]. For NYU Depth V2 we follow [26]. We use the first 1000 RGB/depth pairs for training and the remaining 449 pairs for testing, where each low-resolution depth image is generated by applying a nearest-neighbor interpolation. For Sintel, following [25], 908 samples, and 133 samples from clean pass are used for training, and testing, where each lowresolution flow image is generated by applying a bilinear interpolation. Image denoising datasets. We perform denoising experiments also on NYU Depth V2 [51], as well as on BSDS500 [53]. For NYU Depth V2, we use the same split as for upsampling. BSDS500 contains 500 natural images. We train on the training set (200 images) and the validation set (100 images). We evaluate on the provided test set (200 images). Following [54], to train a blind Gaussian denoiser, random Gaussian noise is added to the clean training depth images in NYU Depth V2 and the clean RGB images in BSDS500, with a noise level σ ∈ [0, 55]. For testing, we consider three noise levels, σ = {15, 25, 50}. Thus, three separate noisy test images are generated for each original test image.
Pre-processing. For all datasets, we normalize the input images by scaling their pixel values to the range [0, 1]. According to Eq. (15), the guidance image G and target image I should have the same number of channels. In the depth/RGB dataset, the target is the 1-channel depth image. Thus, rgb2gray operation is used to transform a 3-channel RGB image into a 1-channel grayscale image as guidance. During training, we augment the images by randomly cropping 256 × 256 patches. No cropping is performed during testing.
Network and optimization. The proposed network is optimized in an end-to-end manner. We implement the network with TensorFlow on a machine with a single GTX 1080 Ti GPU. The optimizer is Adam with a mini-batch of 1. We set β 1 to 0.9, β 2 to 0.999, and the initial learning rate to 0.0001. Optimization is terminated after 1000 epochs.  [13] and the weighted guided filter [20] are used for f a , our filter is denoted by "Ours + GF" and "Ours + WGF", respectively. Our filters perform at least as good as the baselines. B. Unsharp-mask guided filtering without learning.
The goal of the first experiment is to validate our formulation as a valid guided filter. Here we do not rely on any deep learning for estimating f a . Instead, we use the function a of the guided filter [13] and the weighted guided filter (WGF) [20] as f a . F L (G) and F L (I) are generated by cascaded box mean filters. Using our formulation as a conventional guided filter, we provide qualitative and quantitative results to demonstrate that our filter performs as good as, or even better than the guided filter [13] and the weighted guided filter [20].
Qualitative results. The first example performs edgepreserving smoothing on a gray-scale image. The second example is about detail enhancement. For both, the target image and guided image are identical. Fig. 3 and Fig. 4 show the results of filtering, where we can see that our filter performs as good as the guided filter [13] in preserving structures. In the third example, we denoise a no-flash image under the guidance of its flash version to verify the effect of structure-transferring. The denoising results of our filter and the guided filter [13] in Fig. 5 are consistent and don't have gradient reversal artifacts.
Quantitative results. Next, we compare our filter with the guided filter (GF) [13] and the weighted guided filter (WGF) [20] for natural image denoising on BSDS500 and depth upsampling on NYU Depth V2. There are two hyperparameters, r and in GF and WGF. Grid-search is used to find the optimal hyperparameters. The results in Table I show that our filter performs at least as good as the guided filter [13] and the weighted guided filter [20], indicating that our formulation makes sense as a guided filter.

C. Unsharp-mask guided filtering with learning
Next, we assess the benefit of our formulation when f a is learned by a neural network. We compare to four baselines: (i) DMSG [23], (ii) DGF [38] (iii) DJF [26], and (iv) SVLRM [27]. The experiments are performed on NYU Depth V2 [51]  e) The smoothing result obtained by the guided filter [13]; f) Our smoothing result. Both our filter and guided filter [13] can preserve good edges while removing noise.    [23]. Compared to DGF, our approach better recovers finer edges as shown in the regions marked by the red boxes, and avoids producing artifacts as shown in the regions marked by the black boxes.
We compare these baselines separately. For each comparison, the network for f a is the same as the network used in the compared method. We use a box mean filter with radius r = 8 to obtain F L (I) and F L (G). For depth upsampling (16×), we first upsample the low-resolution depth image by bicubic interpolation to obtain the target resolution for F L (I). We also use the upsampled depth image as the input of the network, following [26], [27], [38]. One exception is the comparison with DMSG [23] which uses the original low-resolution depth image as the input of the network. Comparison with DMSG [23]. Fig. 6 (a) demonstrates our approach achieves better upsampling results than DMSG [23] in terms of RMSE. DMSG performs depth upsampling based on spectral decomposition. Specifically, a low-resolution depth image is first decomposed into low-frequency components and high-frequency components. The low-frequency components are directly upsampled by bicubic interpolation. The high-frequency components are upsampled by learning two convolutional networks. The first network is used to obtain multi-scale guidance. The other one performs multiscale joint upsampling. The difference between our method and DMSG mainly lies in two aspects. First, we don't use the first network and just use the second network for amount function f a to explicitly perform structure transfer instead of directly predicting the filtered output image. We find that our approach avoids halo effects more successfully, as shown in Fig. 6 (d) and (e). Second, Hui et al. use a Gaussian filter to smooth the low-resolution target depth image when generating its low-frequency components. After that, they upsample the low-frequency components by bicubic interpolation. However, this step is likely to produce artifacts, as shown in Fig. 6 (f). Since the network learning focuses on upsampling highfrequency components, the artifacts still remain in the final upsampling output, as shown in Fig. 6 (d). By contrast, we first upsample the low-resolution target depth image before smoothing. By doing so, the artifacts generated by bicubic interpolation can be removed by smoothing, as shown in Fig.  6 (g).
Comparison with DGF [38]. Wu et al. [38] learn two networks to amend the guidance and target images before feeding them to the guided filter [13]. The learned guidance and target images fit the guided filter [13] better than the original ones. However, DGF still suffers from the halo problem since its final filtering output is generated by the guided filter [13]. Our approach performs better than DGF [38] for both upsampling and denoising tasks, as demonstrated in Fig. 7 (a) and (b). As shown in Fig. 7 (e) and (g), the important edges are unavoidable to be smoothed because the structure-transferring is performed in an undesirable fashion. By contrast, our approach performs better on preserving and transferring important structures, as shown in 7 (f) and (h), as the amount function f a is learned in a pixel-adaptive way through a deep neural network instead of designed manually.
Comparison with DJF [26]. The network in our method directly uses the unsharp masks of the target image and the guided image as inputs, and learns to estimate the amount function f a for explicitly deciding how to transfer the desired structure from the guidance image to the target image. By contrast, the network in DJF [26] uses the original guidance image and target image as inputs, and directly predicts filtered output relying on feature fusion. The implicit structure transfer is likely to cause slow convergence and the unwanted contents to be transferred from guidance image to target image, as shown in Fig. 8. From Fig. 8 (a) and (b), we can see our approach convergences faster and achieves better filtering performance than DJF [26] on both upsampling and denoising tasks.
Comparison with SVLRM [27]. Lastly, we compare to the state-of-the-art in deep guided filtering, namely SVLRM [27]. Here, we analyze two drawbacks of SVLRM [27], as illustrated in Fig. 9 (c-e). First, f α (I, G) and f β (I, G) is likely to learn similar structure information. This is because they share the same training dependencies; such as input, network architecture and objective function. As a result, f α (I, G) can't  [38]. The parameters of the guided filter [13] used in DGF are r = 4, = 0.1 2 . Our learned amount function performs better on structure-transferring than the manually designed one as shown in the region marked by red boxes of (g) and (e). Thus, our filter reduces the oversmoothing of important edges as shown in the region marked by black boxes of (e) and (f).
transfer the desired structure from guidance G to the output image of f β (I, G). Second, SVLRM behaves like DJF [26] when the filtering performance is determined by f β (I, G). The implicit joint filtering causes slow convergence and the unwanted structures are transferred. By contrast, our approach focuses on estimating the amount function f a for explicit structure transfer, leading to more desirable filtering results, as illustrated in Fig. 9 (h-j). Fig. 9 (a) and (b) demonstrate the better performance of our approach compared to SVLRM [27], for both upsampling and denoising. Amount function f a . When we estimate the amount function f a through a convolutional neural network, the network architecture plays an important role in filtering performance. We have compared our filtering formulation with several baselines when f a is estimated by different networks used in these baselines. Generally, we found deep networks perform better than shallow networks, e.g., the network of SVLRM with a depth of 12 achieves an RMSE of 7.23 for upsampling (16×) and a PSNR of 40.27 for denoising (σ = 50), better than the RMSE of 8.19 and the PSNR of 39.59 obtained by the network of DJF with a depth of 6. One explanation for this is the fact that the deep network has more ability to express complex functions than shallow ones. In Eq.(15), we use the unsharp masks of the target image and guidance image as the input of the amount function network f a . The raw target image and guidance image can also be the input. Next, we perform an experiment to study which input performs better. The network of DJF [26] is used for f a . On NYU Depth V2, using the unsharp mask as input achieves an RMSE of 8.19 for upsampling (16×) and a PSNR of 39.59 for denoising (σ = 50), better than the RMSE of 8.64 and the PSNR of 39.31 obtained by using the raw image as input. We find that using the unsharp mask as input not only achieves better filtering performance, but also convergences faster because network learning can focus on extracting the desired structure without the interference of redundant signals from the smooth basis of the image.
Smoothing filter F L . To obtain the unsharp masks G m and I m , we need a smoothing filter for F L (I) and F L (G). Next, we explore how the smoothing process affects the final filtering performance, we compare three different smoothing filters with different hyper-parameters on NYU Depth V2 for  Fig. 9: Comparison with SVLRM [27].
In SVLRM, f α and f β are likely to learn the same structure information. In this case, f α can't transfer the structure desired by f β , resulting from over-smoothing of edges as shown in the regions marked by the blue boxes of (d) and (e). When the filtering output is determined by f β , SVLRM behaves like DJF [26], causing the transfer of unwanted contents from guidance image to target image as shown in the regions marked by the red boxes of (c-e). By contrast, our approach resolve the problems of SVLRM by explicitly learning structure-transferring, leading to more desirable filtering results as shown in regions marked by the blue and red boxes of (h-j).
Filter size r/ < 16× depth upsampling. The hyper-parameter is the filtering size r for the box filter, or the Gaussian variance σ for the Gaussian and bilateral filters. We use three different values: (4,8,16). The network of DJF [22] is used for f a . As shown in Fig. 10, our method is robust across filter type and size. We opt for the box filter with a filtering size of 8 throughout our experiments because it is simple, efficient and effective.

D. Successive filtering network
Next, we investigate the effect of the network we designed for our successive filtering formulation. There are multiple amount blocks used in the successive filtering network. We explore how the number of amount blocks L affects the filtering performance on NYU Depth V2 for depth upsampling (16×) and depth denoising (σ = 50). F L is a box mean filter with radius r = 8. For depth upsampling (16×), we first upsample the low-resolution depth image by bicubic interpolation to obtain the target resolution for F L (I) and network learning. We make two observations from the results shown in Table II. First, our model's filtering performance is consistently improved when increasing L from 1 to 5. Second, we can obtain multiple (L) filtering results by training a single network. Each filtering result is as good as the result obtained by an independently trained network. The model's filtering performance doesn't improve a lot when L is increased from 4 to 5. Thus, we opt for L = 4 in the following experiments.

E. Performance analysis.
Next, we analyze the performance of different deep guided filtering methods from three aspects: run-time performance, model parameters and filtering accuracy. For our methods, we use the successive filtering network with L = 4. Thus, we can obtain four filtering models by training a single network, indicated by Ours(Î (1) ), Ours(Î (2) ), Ours(Î (3) ) and Ours(Î (4) ). We perform upsampling (16×) with all methods on the testing datasets (499 RGB/depth pairs) of NYU Depth V2. We perform all the testings on the same machine with an Intel Xeon E5-2640 2.20GHz CPU and an Nvidia GTX 1080 Ti GPU. The average run-time performance on 499 images with the size of 640 × 480 is reported in GPU mode with TensorFlow. From Table III, we can see that Ours(Î (1) ) has the fewest parameters (17 k), and Ours(Î (4) ) achieves the best filtering accuracy (6.07 RMSE). Ours(Î (1) ) achieves a competitive average run-time performance (31 ms) compared to the best one achieved by DJF [22] (29 ms).

F. Depth and flow upsampling
Tables IV and V show results for upsampling a depth image or optical flow image, under the guidance of its RGB image. We have noted that the existing works use different training settings and evaluation protocols. For fair comparison, we reimplement the main baseline methods under our experimental settings. Our filter performs well, especially on Sintel and the larger upsampling scales on NYU Depth. Different from the related works [23]- [27], [38], our model learns an amount function f a to explicitly decide how to transfer the desired structure from guidance image to target image. Thus, our model can be more effective and efficient to learn the desired output. Fig. 11 show our ability to better recover finer edges.

G. Depth and natural image denoising
Our formulation also allows for standard filtering without a guidance image by simply making G identical to I in Eq. (17). Intuitively, such a setup defines a structure-preservation filter, while the guided variant defines a structure-transfer filter. Next, we evaluate the ability to remove Gaussian noise from depth and natural images. For depth image denoising, its RGB image is used as guidance. For natural image denoising, the target and guidance image are the same RGB image. The quantitative results are shown in Tables VI and VII. We obtain the best PSNR and SSIM scores on both datasets for all three noise levels. Fig. 12 and Fig. 13 highlight our ability to better remove noise and preserve finer edges compared to other filters.

H. Cross-modality filtering
Finally, we demonstrate that our models trained on one modality can be directly applied to other modalities. Here, we use the models trained with RGB/depth image pairs for the joint upsampling of bw/color and RGB/saliency image pairs, and the joint denoising of RGB/NIR and flash/no-flash (a) Guidance (b) Ground truth (c) DMSG [23] (d) DGF [38] (e) DJF [26] (f) SVLRM [27] (g) PAC [27] (h) Ours Fig. 11: Optical flow upsampling (16×) on Sintel. We are able to maintain sharp and thin edges.
Following [26], for the multi-channel target image, i.e., no-flash image, we apply the trained models independently for each channel. For the single-channel guidance image, i.e., NIR   image, we replicate it three times to obtain a 3-channel guidance image. Joint upsampling. To speed up the translation from one image to another image, one strategy is to perform translation at a coarse resolution and then upsample the low-resolution solution back to the original one with a joint image upsampling filter. Here, we demonstrate that our models act as joint upsampling filters well on bw/color and RGB/saliency image pairs translation tasks. For bw/color translation, we use 68 bw images from BSD68 dataset [56], and the colorization model proposed by Lizuka et al. [57] is used as translation model. For RGB/saliency translation, we use 1000 RGB image from ECSSD dataset [58], and the saliency region detection model proposed by Hou et al. [59] as translation model. The input images, i.e., bw images and RGB images, are first downsampled by a factor of 4× using nearest-neighbor interpolation, and then are feed into the translation models to generate the output images. After that, we recover the output images to the original resolution under the guidance of the original input images by various joint upsamling methods. Table VIII shows the quantitative results, and we can see that our model achieves the best performance for both two tasks. The joint upsampling pipeline performs more than two times faster than direct translation on the GPU mode. We also provide the qualitative results in Fig. 14 and 15 to show that the proposed model better recovers finer details.
Joint denoising. We introduce two datasets for the joint denoising experiments. Flash/no-flash [60] consists of 120 image pairs, where the no-flash image is used for denoising under the guidance of its flash version [13], [26], [61], [62]. Nirscene1 [63] consists of 477 RGB/NIR image pairs in 9 categories, where the RGB image is used for denoising under the guidance of its NIR version [26], [62], [64]. Table IX shows that our filter has a better denoising ability than alternatives. Fig. 16 and Fig. 17 provide qualitative results, which shows that our model better preserves important structures while removing noises.
VI. CONCLUSION In this paper, we have introduced a new and simplified guided filter. With inspiration from unsharp masking, our proposed formulation only requires estimating a single coefficient, in contrast to the two entangled coefficients required in current approaches. Based on the proposed formulation, we introduce a successive guided filtering network. Our network allows for a trade-off between accuracy and efficiency by choosing different filtering results during inference. Experimentally, we find that the proposed filtering method better preserves sharp and thin edges, while avoiding unwanted structures transferred from guidance. We furthermore show that our approach is effective for various applications such as upsampling, denoising, and cross-modal filtering. (c) Ground truth [57] (d) GF [13] (e) SVLRM [27] (f) This paper (c) Ground truth [59] (d) GF [13] (e) SVLRM [27] (f) This paper (e) SVLRM [27] (f) This paper (e) SVLRM [27] (f) This paper


Visual Servoing of Unknown Objects for Family Service Robots

Aiming at satisfying the increasing demand of family service robots for housework, this paper proposes a robot visual servoing scheme based on the randomized trees to complete the visual servoing task of unknown objects in natural scenes. Here, “unknown” means that there is no prior information on object models, such as template or database of the object. Firstly, an object to be manipulated is randomly selected by user prior to the visual servoing task execution. Then, the raw image information about the object can be obtained and used to train a randomized tree classifier online. Secondly, the current image features can be computed using the well-trained classifier. Finally, the visual controller can be designed according to the error of image feature, which is defined as the difference between the desired image features and current image features. Five visual positioning of unknown objects experiments, including 2D rigid object and 3D non-rigid object, are conducted on a MOTOMAN-SV3X six degree-of-freedom (DOF) manipulator robot. Experimental results show that the proposed scheme can effectively position an unknown object in complex natural scenes, such as occlusion and illumination changes. Furthermore, the developed robot visual servoing scheme has an excellent positioning accuracy within 0.05 mm positioning error.

Introduction
Accurate positioning of objects is the crux to real-world applications of family service robots. Obviously, the basic function of service robots is that it can operate, grasp and move a specific object selected by the user freely. The first step to implement the above task is to make a robot positioning an object with high accuracy and strong robustness. At present, most visual servoing approaches, which have been widely and successfully used for applications where the object to be manipulated is known beforehand [1][2][3][4] or the scene is known beforehand [5]. Unlike industrial robots, family service robots usually work in highly unstructured environments [6,7], and need more intelligence than industrial robots to perform given tasks [8]. In this environment, the object may be "unknown", these methods need to acquire prior knowledge of the properties of these non-rigid bodies in advance. As a result, they are unable to handle unknown objects.
In view of the complexity of non-rigid body model establishment, some researchers have recently proposed methods based on model-free approaches [18][19][20][21][22][23]. The main idea of these methods is representing the target through artificial marking, which simplifies the extraction of image feature. However, these methods are labor intensive and not efficient. With the development of deep learning approaches, some researchers have proposed to use deep learning algorithms [19,20] to identify the category information of unknown objects and obtain the best grasping position of the object. However, these methods need to obtain the prior knowledge of the category information of each unknown object by off-line learning, and do not consider the problem of robot visual servoing control of unknown objects in complex natural scenes (such as occlusion and illumination changes). As a result, these methods fail to apply for natural scenes.
In this work, we propose a robot visual servoing scheme based on the randomized trees with an effort for avoiding above-mentioned shortcomings. The main objective and feature of the proposed approach are to complete the visual servoing task of unknown objects in various complex natural scenes towards more practical solutions and applications. As opposed to prior work which uses prior knowledge of objects to establish their model [13][14][15][16][17], the proposed method in this work does not need any prior information on object models, such as template or database of the object. User can randomly select an object to be manipulated prior to the visual servoing task execution. Furthermore, inspired and improved from some deep learning algorithms [19,20], the proposed method can perform well in complex natural scenes (such as occlusion and illumination changes), which is common in manipulation domains.
The main contributions and novelty of this work can be outlined as follows: & A visual servoing method based on the randomized trees for unknown objects in natural scenes is proposed. It does not need to acquire any knowledge of the object template and its natural scene in advance, and there are always many objects on the robot operating platform during the positioning process. "Unknown Objects" are randomly specified according to the needs of users, then before the robot servoing task starts, the required data only can be acquired on-line for the specified object, that is, the prior knowledge of the geometric model of the object is not needed in advance. & The proposed scheme has been tested and evaluated on a real MOTOMAN-SV3X six degree-of-freedom (6-DOF) manipulator robot. Experimental results show that the developed scheme can effectively position an unknown rigid object or non-rigid object in many challenging nature scenes with occlusion and illumination changes and with a high positioning accuracy.
The remainder of this paper is organized as follows. After discussing related work in Section 2, a brief description on the system structure is given in Section 3. The specific implementation of the three main parts of the system, including the construction of the randomized tree classifier, the computation of the current image features, and the design of the visual controller, is described in detail in Section 4. Detailed experimental results and analyses are provided in Section 5. Finally, summary and future works are presented in Section 6.

Related Work
Research on the visual servoing of objects (including rigid bodies and non-rigid bodies) can be roughly classified into model-based methods and model-free methods. In modelbased methods, the core of these methods is to build the model of the object and estimate its parameters [11, 13-17, 24, 25]. Gratal et al. [11] propose a virtual visual servoing based on saliency map to achieve the visual servoing of unknown objects in natural scenes. Using the approaches in [25,26], a robot can pick up or place a random object on the desktop, but it cannot operate on a specified object and can only remove all objects one by one on the desktop, which would possibly make the ornament to be cleaned up mistakenly. Jadav et al. [15] used a comprehensive dynamic equation to express the movement of the system to manipulate the deformed object, and then used multiple manipulators (or a claw with multiple fingers) to change the shape of the deformed object. The movement of each manipulator proposes an optimization scheme. This paper verifies the effectiveness of the proposed method through two sets of simulation experiments. However, this method heavily relies on model establishment and parameter estimation for non-rigid body, so it is not suitable for unknown objects.
In model-free methods, the key is the representation of targets. Classical methods represent objects through artificial marking [18][19][20][21][22][23]. Using the approach in [18], a robot can complete the task of picking up and placing the unknown object by marking the grasping position of the specified object in advance. But the approach is very difficult to apply to realistic unknown natural environments in the family services. Newer methods address the challenge by using deep learning algorithms. A deep learning algorithm [19,20] is used to recognize the category information of unknown objects and obtain the optimal grasping position of the objects. The approach does not need to obtain the geometric model information of the object prior to the visual servoing task execution.
However, the approach needs to acquire the prior knowledge of the category information of each unknown object by offline learning, and does not consider the problem of the robot visual servoing control for the unknown object in complex natural scenes (e.g. occlusion and illumination changes).
In this work, to solve the problems of model-based methods and model-free methods, we propose a robot visual servoing scheme under unknown objects in various complex natural scenes. Experimental results show that the proposed scheme can effectively position an unknown object in complex natural scenes with strong robustness to occlusion and illumination variations and small positioning error within 0.05 mm. It is showed that the proposed novel visual servoing scheme can further improve flexible operations of the visual servoing.

Problem Description and Overview of System Structure
Robot visual servoing is intended to control the relative pose of robot and object using the visual information. It can allow the robot to work in dynamic and uncontrolled environments [26]. There are two kinds of robot visual servoing structures, including position-based visual servoing (PBVS) and imagebased visual servoing (IBVS). This paper adopts the imagedbased visual servoing structure and eye-in-hand configuration. In this configuration, the camera is mounted on the robot endeffector so that it could be moved along with the robot. The relative pose of the robot and object is represented as the difference between the current image features and the desired image feature, then the task of robot visual servoing can be defined as minimizing the relative pose of the robot and object. In other words, under this eye-in-hand configuration and visual servoing structure, both robot visual positioning and visual tracking can be viewed as a positioning problem in an image feature space. Therefore, how to detect an object in complex scenes, to calculate the feature of object in the current image, and to design visual controller are three key issues for achieving a robot visual servoing mission in natural scenes.
To solve the above problems, this paper proposes a robot visual servoing scheme based on the randomized tree classifier in natural scenes. The overall structure of the proposed scheme is shown in Fig. 1. The system mainly includes three parts as follows: building a randomized tree classifier, computing the current image feature, and designing the visual controller. The basic idea of the algorithm is: Firstly, user randomly selects an object to be manipulated prior to the visual servoing task execution. Then, the raw image about the object can be captured by the camera and used to generate a number of sample data sets for building the randomized tree classifier. Secondly, the current image features, which are represented as 2D pixel coordinates of the object image centroid, can be computed using the previously built classifier. Finally, visual control input can be calculated according to the image feature error and is applied to the robot to achieve the robot visual positioning for unknown objects. In Fig. 1, f d represents the desired image features and f denotes the current image features (as shown by the red circle).
The basic principle of the three main parts of the servoing system will be stated individually in detail in next section.

Proposed Solution
In this section, detailed implementation of all functions highlighted in Fig. 1 will be described according to the order listed above, which includes the construction of the randomized tree classifier, the computation of the current image features, and the design of the visual controller.

Construction of the Randomized Tree Classifier
The main function of this module is to construct a randomized tree classifier for recognizing and detecting unknown objects randomly selected by the user. The whole process is shown in Fig. 2, which can be further divided into the following four steps.
Step 1. Selecting the unknown object to be manipulated.
Firstly, the object selected randomly by the user will be put on the training station whose background is clean without any clutter, and the robot is also moved to the training station. Then, the desired image can be obtained through segmenting the current image captured by the camera mounted on the robot's end-effector. If the object is a 3D non-rigid body, its rough 3D model can be obtained through the ImageModeler software, and then the 3D object image will be obtained using the above method.
Step 2. Preprocessing. The main purpose of the preprocessing is to get the gray image of the desired image.
Step 3. Extracting the object features and generating the view sets. The stable affine invariant features are extracted on the preprocessed image. Then patches, whose center is each feature point, are obtained. These patches will form view sets of image feature points of the object, or object view sets for short.
Step 4. Establishing the randomized tree classifier. A randomized tree classifier can be built using these object view sets.
The detailed implementation of feature extraction, object view sets generation and randomized tree classifier establishment are described as follows.

Feature Extraction
Characteristics of image features used in the control loop, especially for the visual servoing structure based on images, will directly affect the stability and robustness of a robot visual servoing system, and this is one of the important factors for robotic systems to be successfully applied to complex environments. Therefore, this paper adopts a two-level feature extraction method to obtain affine invariant stable features. Firstly, LOG-FAST feature extraction method is used for rapidly extracting features from the gray image of the raw object. The LOG operator is utilized to conduct Gaussian filtering and image sharpening processing so as to eliminate noises of the image. Then FAST-9 operator is used to extract the feature in different scale spaces.
To further obtain affine invariant stable features, it is necessary to conduct selection operation after the preliminary extraction. The main steps to achieve above task can be stated as follows: Step 1. Generating M new images by conducting affine transformation on the grayscale image. Affine transformation is a combination between non-singular linear transformation matrix A¼R θ R −1 φ SR φ and translation matrix t = [t x , t y ] T , where R θ and R φ are the rotation matrices that correspond to θ and φ respectively, and they are within the ranges of [-π, π]. S = diag {λ 1 , λ 2 } represents the image scaling transformation matrix, and λ 1 , λ 2 are within the range of [0.2, 1.5]. Besides, t x , t y are within the range of [0, 2]. M new images are generated by randomly selected parameters θ, φ, λ 1 , λ 2 , t x , t y , and white noise is added into the generated M new images.
Step 2. Determining affine invariant stable features. Firstly, LOG-FAST feature extraction method is adopted to Camera X e u  Fig. 2 Process of constructing a randomized tree classifier extract the features on the M new images. Then, the inverse transformation is utilized to recover the extracted feature. Finally, the successful matching frequency between the recovered features and the features of the original image is calculated. The features with the top N frequency are considered as the "stable" features, which are illustrated with red circles in Fig. 3.
These features construct a feature set K = {k 1 , k 2 , …, k N }, where 1 to N are ultimately determined stable features. Each feature is tagged to denote a class, and the different classes are indicated as C = {c 1 , c 2 , …, c N }. Then these stable features are used to construct view sets to be employed for building the randomized tree classifier.

Establishment of View Sets
When capturing a frame of new image in the robot visual servoing process, the most critical problem is to determine whether the current image involves the object and object's location is in the image. The first step for solving the above problem is to detect the features on the object. So, the view set of each feature needs to be further established after the stable features are obtained. Feature patches are extracted on the M affine transformed images, and the size of each patch is 32*32. The view sets consist of all the extracted feature patches, and the size of view sets is M*N finally. All feature patches with the same number will form a small collection, thus for the N stable features, N small sets are constructed: V n = {v n1 , v n2 , ⋯, v nm }, 1 ≤ n ≤ N,where each V n set includes m elements, and it is the view set of a feature. Figure 4 shows the view set of a certain feature, where different elements indicate the different locations for the same feature in different perspectives.

Establishment of Randomized Tree Classifier
Binary decision tree is adopted in this system, which has only one root node and two child nodes. Each node is divided into two child nodes and the other nodes follow recursively until the bottom node has no branch. The bottom node is named as leaf node. The view sets are put into the root node, and patches of each view set traverse from the root to the leaf. In the traversal, every node will test the patch to determine whether this patch belongs to a certain node. This system adopts a gray image, thus the information gain of the gray information feature is the largest. Gray information is selected as the criterion of classification in this paper. The discriminant of each node can be written as: where τ is the default threshold, I(p, m 1 )and I(p, m 2 ) represent the gray values of two pixels which are randomly selected in the patch p entering the tree. When all randomly selected patches enter the randomized tree, the number of each class's patches entering the leaf node m and the number of all patches entering the leaf node M should be calculated. The ratio between m and M is the posteriori probability that a certain feature is identified as a certain class. If the number of the ith feature class's patches entering the leaf node is m i and the number of all the patches entering the leaf node is M, then the posterior probability that the ith feature is identified as the ith feature class can be represented as: where η represents the leaf node. The leaf node stores the posterior probability P η(l, p) (Y(p) = c) of each class. In general, as the size of the sample set is large, it is difficult to guarantee a high accuracy result using one randomized tree classifier. Statistically speaking, more randomized tree classifiers can compensate this shortcoming. Therefore, the method of establishing more trees to speed up the process of the object detection is adopted. The next step is to use the built randomized tree classifier to detect objects and to compute the current image features on the current image captured by the camera in the natural scenes in real time.

Computing the Current Image Feature
The main function of this module is to detect the object in the current image and to compute the features of the current image. The whole process can be shown in Fig. 5.
where P η(l, p) represents the posterior probability of the patch p entering the leaves nodes of the lth tree T l , c is the feature class, p c (p) is the mean posterior probability of the class c and D c is the default threshold (60% used in the later experiment). When the posterior probability is larger than D c , the feature of this patch is classified as class c, and further, the center of this patch matches the feature of the class c in the desired image. Otherwise the patch belongs to the background or a misclassification and should be discarded. Thus, the match between the current features and the cth class features is implemented. When the number of matching reaches a certain threshold, it is determined that the current image contains the object. In order to calculate the current image feature, the RANSAC algorithm is used to estimate the homograph matrix H. Then H is used to calculate the centroid coordinate of the object, which is the current image features. In Fig. 5, the centroid of an object image is represented by a red circle as shown in the picture named "current feature".

Design of the Visual Controller
After obtaining the current image features, one can further design the visual controller according to the image error to drive the robot for positioning an unknown object in natural scenes. The image error e can be defined as: where f d and f are the desired image features and current image features, respectively. For an image-based visual servoing structure used in the paper, image Jacobian matrixJ im , as shown in Eq. (5), is often used to describe the relationship between the changes of the features in the image spaceḟ and spatial velocity of the robot end-effector in the robot workspaceṙ.
whereṙ is spatial velocity of the robot end-effector, including linear velocity and angular velocity. In other words,ṙ is the calculated visual control input u to be applied to the robot end-effector;ḟ can be viewed as the image feature error e. Then, a simple visual control law u based on the inverse Jacobian matrix can be designed as shown in Eq. (6).
where [υ T ω T ] T is the camera's (or robot's) velocity, and J im + is the pseudo inverse of image Jacobian matrix, and K p is the proportional control gain. The Laypunov function is defined as V = 1/2e T e, V > 0. Then, the derivation of the Laypunov functionV can be described as: For eye-in-hand configuration used in the paper, camera is mounted on the robot end-effector and it moves along with the robot. So, in this camera-robot configuration, both robot visual positioning and visual tracking are considered as the positioning problem in the image features space. Therefore,ḟ d ¼0. Then, For robot visual control input in any ith directionu i , it can easily prove that the Laypunov function satisfiesV i < 0 by using Eq. (8). Therefore, the closed-loop system is asymptotically stable according to the Lyapunov stability theory, and the robot can be controlled to move to the desired image features.

Experimental Results
Five robot visual positioning experiments are conducted, mainly including visual servoing of 2D rigid objects and non-rigid objects in several natural scenes, to validate the performance of the proposed robot visual servoing scheme for unknown objects. Figure 6 shows the experimental platform for robot visual servoing, which contains a CCD camera, a MOTOMAN SV3 6-DOF manipulator robot and a personal computer with Windows operating system and the OpenCV 2.4.1 software. The CCD camera is mounted on the robot hand claw. It moves following with the motion of the robot hand claw, while its internal and external parameters are unknown. The image plane size is 1024*768 (unit: pixel), so the desired image feature f d is the center of the image plane coordinate (512, 384) (unit: pixel). In order to evaluate the performance of the proposed robot visual servoing scheme in the real world, two types of five robotic visual positioning experiments were performed in the complex nature scenes, including unknown rigid objects and unknown non-rigid objects.
The visual control law shown in Eq. (6) is used, and the specific values of the relevant parameters are:K p = diag (0.04,0.04).
The maximum joint running speed of Motoman SV3 6-DOF robot used in the experiment can reach 300 o /s. In the experiment, for the sake of safety, the running speed of the robot is set at a low value of 10 o /s. In Windows operating system, average image capturing and processing time per frame is about 0.156 s in the experiment, which can meet the real-time control requirements of the arm robot at low speed. The average processing time of the key components (taking the initial frame images captured in experiment 5.1.1 as an example, the average processing time of the key components for different initial frame images captured in each visual servoing task is basically unchanged) is as follows:

Robot Visual Servoing of 2D Rigid Objects in Natural Scenes
The purpose of this experiment is to verify the performance of the proposed approach for unknown 2D rigid objects in complex nature scenes, including cases with no-occlusion, with occlusion and with illumination change. The detailed results are provided in Experiments 1-3.

Visual Positioning of the Unknown Object without Occlusion
Visual positioning results are shown in Fig. 7. Fig. 7a illustrates the initial frame, where the center coordinate (512, 384) of image plane is visualized by a red cross. Namely, it is the desired image featuref d . The object is represented by a green rectangular bounding box. The object centroid is visualized by a blue circle and it represents the current image featuref(k) = (u, v) T . The positioning results of middle frame and end frame are illustrated in Fig. 7b and c, respectively.

Visual Positioning of the Unknown Object with Occlusion
Visual positioning results are shown in Fig. 8. The positioning results of middle frame and end frame are illustrated in Fig. 8b and c, respectively. During the visual positioning, the object is occluded by the black Mobile Hard Disk. The experimental result shows that the robotic system still can detect the object and finish the positioning in spite of the large occlusion.

Visual Positioning with Illumination Variations
The purpose of this experiment is to verify the availability of the proposed approach under the scenarios containing overall and local illumination variations. Visual positioning results are shown in Fig. 9. Initial illumination is shown in Fig. 9a. During the visual positioning, the case of turning off the light in the robot workplace is shown in Fig. 9b, then add a beaming light as shown in Fig. 9c, then remove a beaming light as shown in Fig. 9d. Visual positioning results under the above three illumination variations are illustrated in Fig. 9 b and 9d, respectively. It can be seen from Fig. 9 that robot can successfully implement the visual servoing of unknown 2D objects in natural scene in spite of large illumination variations.

Robot Visual Servoing of Non-rigid Objects in Natural Scenes
In order to further verify the effectiveness of the proposed approach for non-rigid objects in natural scenes, another two

Visual Positioning of 2D Non-rigid Objects
The purpose of this experiment is to verify the effectiveness of the proposed approach for 2D non-rigid objects in natural scenes. A sponge is selected as a 2D non-rigid object. The experiment is designed to simulate some practical applications for service robots, such as folding the clothes and operating the soft tissue in surgery and so on. During the visual positioning, the basic shape of the sponge has undergone tremendous changes. The sponge is performed with up and down mixture non-rigid deformation and visual positioning results are shown in Fig. 10. At the very beginning of the servoing, the sponge is squeezed up and its deformation is shown in Fig. 10a. Then sponge is squeezed down and visual (a) Initial frame (b) Middle frame (c) End frame  positioning result is shown in Fig. 10b. After Fig. 10b, the sponge is folded up as shown in Fig. 10c until the end of visual positioning process as shown in Fig. 10d. It can be seen from Fig. 10 that the robot can still successfully implement the visual positioning of the 2D non-rigid object.

Visual Positioning of 3D Non-rigid Objects
The purpose of this experiment is to verify the capability of the proposed approach for 3D non-rigid objects. A plush toy is selected as a 3D non-rigid object. As the plush toy has 3D  characteristics, it is necessary to first build its rough 3D model by ImageModeler software using multiple views images of the plush toy during the visual positioning. The plush toy is then performed with certain non-rigid deformation and its pose is also changed. Visual positioning results are shown in Fig. 11. Initial pose and shape of the plush toy is shown in Fig. 11a and plush toy does not have any deformation. During the visual positioning, plush toy is squeezed and visual positioning result is shown in Fig. 11b. Then the pose of plush toy is changed and visual positioning result is shown in Fig. 11c. Continuing to be squeezed until the end of positioning as shown in Fig. 11d, it can be seen from Fig. 11 that the robot can still successfully implement the visual positioning of such a 3D non-rigid object.

Further Analysis and Discussion
Positioning trajectory curves in the different directions and different spaces can provide more insight into the visual servoing behavior. During the visual positioning, pose of the object can be changed freely. In this experiment, pose of the object is changed at the late servoing when the robot is nearby the desired position. Visual positioning results are shown in Fig. 12. The positioning trajectory of object is shown in Fig. 12a in the image plane u and v, and the positioning trajectory of robot is shown in Fig. 12f in the robot workspace x and y. Positioning error curves in the image plane, u and v directions are shown in Fig. 12b, c and d, respectively. It can be seen from Fig. 12 that the proposed visual servoing scheme has a better convergence performance. Table 1 lists the mean, standard deviation (std) and max of the robot positioning error in 10 different poses of the 2D object without occlusion. The "max" represents the ratio between the max absolute of 10 times positioning error and the motion range of this direction. The model of CCD camera used in the experiment is MV-VS078FC-L, the image resolution is 1024*768, and the corresponding pixel size is 4.65μm*4.65 μm, that is, the physical size of each pixel in x and y directions is 0.00465 mm. It can be seen from Table 1 that the positioning error in x and y directions are about 0.004 mm and 0.048 mm respectively and the largest absolute relative error and standard deviation are also very small. Therefore, the developed positioning system has high accuracy with a positioning error within e ¼ ffiffiffiffiffiffiffiffiffiffiffiffiffiffi e 2 x þ e 2 y q ¼ ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 0:004 2 þ 0:048 2 p ¼ 0:05mm.
As showed in Table 1, the five positioning experimental testing results show that the proposed approach of robot visual servoing based on randomized tree classifier can implement visual positioning of unknown objects in complex natural scenes with occlusion and illumination variations.

Conclusion and Future Works
This paper proposes a robot visual servoing scheme to locate the robot to unknown objects in natural scenes with family services as application scenario. Five visual positioning experiments for unknown rigid object and non-rigid object in various nature scenes are conducted on a MOTOMAN-SV3X six degree-of-freedom manipulator robot. Experimental results show that the proposed scheme can effectively position an unknown object in complex natural scenes with strong robustness to occlusion and illumination variations and small positioning error within 0.05 mm. Furthermore, the system does not need any template nor any database of the object prior to the visual servoing task execution. Once the object is selected by the user freely, all the needed data can be obtained online and the robot can complete the positioning task automatically.
However, current method cannot position the unknown multiple objects at the same time. User can only specify one object for visual servoing task before performing the visual servoing task. If the current visual servoing task contains multiple targets, we have to reuse our method, which will become more time-consuming. In other words, our method is only suitable for the case where the number of targets is small and is not suitable for the case where the number of targets is too large.
As future works, the proposed robot visual servoing scheme will be further extended into two aspects: 1) to position the unknown multiple objects by combining multiple objects recognition and detection approaches [27,28]; 2) to autonomously grasp the unknown object by combining deep reinforcement learning in view of its strong learning capability. Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.