extfieldsofstudy:
Medicine
Computer Science

text:
Extracting Factual Min/Max Age Information from Clinical Trial Studies

Population age information is an essential characteristic of clinical trials. In this paper, we focus on extracting minimum and maximum (min/max) age values for the study samples from clinical research articles. Specifically, we investigate the use of a neural network model for question answering to address this information extraction task. The min/max age QA model is trained on the massive structured clinical study records from ClinicalTrials.gov. For each article, based on multiple min and max age values extracted from the QA model, we predict both actual min/max age values for the study samples and filter out non-factual age expressions. Our system improves the results over (i) a passage retrieval based IE system and (ii) a CRF-based system by a large margin when evaluated on an annotated dataset consisting of 50 research papers on smoking cessation.

Introduction
Clinical trials are an important source of scientific evidence for guiding the practice of evidencebased medicine. However, many characteristics of clinical trials are only reported in the published research articles. The health service community could benefit from knowledge bases populated with detailed information from clinical trials reported in research articles. With this in mind, clinical information extraction aims to extract such information from journal articles that report randomized controlled trials (Kiritchenko et al., 2010;Wallace et al., 2016).
Relevant information about clinical trials can be categorised along: (i) trial's population characteristics (e.g. minimum and maximum age of the participants, education level, marital status, health status), (ii) intervention methods, both what is being done (e.g. specific drug and dosage, planning sessions, use of an app for daily reporting) and how it is being administered (e.g., where, how often and by whom), and (iii) outcome of the study (e.g., 30% of the population stopped smoking after 6 months).
In this paper, we focus on extracting population characteristics and in particular minimum and maximum (min/max) age values associated with the study samples from clinical trials research articles.
Unlike (Summerscales, 2013), our aim is to extract information from the full article, rather than only from the abstract, as we have observed that age information is not always described in the abstracts. In our testing dataset consisting of 50 research papers, only nine papers describe the min/max age information in their abstracts.
Naturally, analysing the entire article presents many challenges. Our goal is to identify the factual min/max age value information for the persons who actually participated in the clinical trial (see Example 1 and Example 2 below). This should be distinguished from non-factual min/max age information (Example 3 and Example 4) and also from min/max age information which is not related to the participants in the study (Example 5 and Example 6).
(2) participants aged 18-24 years were randomized to a brief office intervention (n=99) or to an expressive writing plus brief office intervention (n=97).
(3) To be included in the study, smokers had to be between the ages of 18 and 60 years . . .
(4) The subjects were eligible for inclusion if they were at least 18 years of age, reported smoking 10 or more cigarettes per day, . . .
(6) Smoking Dutch youths had in many cases tried their first cigarette at the age of 11-12 years.
Our proposed system extracts factual min/max age values of the study samples directly from research articles in PDF format. We leverage the massive structured clinical study records from ClinicalTrials.gov to provide distant supervision for min/max age value extraction. Furthermore, inspired by the work on hedging detection on Bioscience domain (Light et al., 2004;Kilicoglu and Bergler, 2008;Farkas et al., 2010), we explore a list of "speculation cues" to filter out non-factual min/max age expressions. Our system improves the results over (i) a passage retrieval based IE system and (ii) a CRF-based system by a large margin when evaluated on an annotated dataset consisting of 50 research papers on smoking cessation.

Clinical Information Extraction
In general, research on information extraction from medical literature is still in its infancy involving a number of limitations, such as lack of common benchmarking datasets, and a lack of general consensus on the class of approaches that are reported to work well on such benchmarks. Some work has been conducted on supervised approaches for medical information extraction. Multiple studies have concentrated their efforts on medical abstract. In (Kim et al., 2011), the authors propose a conditional random filed (CRF) classification method for labelling medical abstract sentences according to medical categories, such as outcome, intervention, population. Hansen et al., 2008(Hansen et al., 2008 developed a Support Vector Machine algorithm for extracting the number of trials participants from medical abstracts, while in (Hassanzadeh et al., 2014), the authors use a machine learning approach for classifying abstract sentences according to the PICO (Population, Intervention, Comparison, Outcome) scheme.
Other studies have exploited the entire article, for the extraction of papers' metadata as (Lin et al., 2010): the authors propose a preliminary system based on CRF for extracting formulaic text (authors names, email and institution) as well as some key study parameters as free text, from PubMed-Central articles. They reach promising results for the formulaic text, but only moderate success for the free text attributes. The study in (Luan et al., 2017) involves finding key-phrases from scientific articles and then classifying them. However, these categories are much broader (coarse-grained), e.g. 'process', 'task' etc., than the fine-grained categories in our task (min/max age).
A few studies have tackled the min/max age extraction problem. Most research work on extracting information from clinical trial literature considers "eligibility criteria" as a target element, which often contains min/max age information (de Bruijn et al., 2008;Kiritchenko et al., 2010).
However, min/max age information contained in the eligibility criteria refers to the planned min/max age and may be different from the actual min/max age values of the study samples (for example: the researchers could decide to test a population of women between 20-30 years, but realistically they could gather participants only between 22 and 28 years old). (Summerscales, 2013) carefully designed a number of heuristic rules to extract min/max age values of the study population from the abstracts. We differ from this latter work as we (a) extract such information from the full articles and (b) use a machine learning approach. In addition, we integrate the rules designed by (Summerscales, 2013) into our passage retrieval based IE system as a baseline.
Generally, in contrast to previous work, in this paper we a) concentrate on the extraction of population characteristics, b) use the entire article for detecting the min/max age and c) compare an unsupervised approach with a QA-based approach.

Question Answering
Most recently, reading comprehension or question answering based on context has gained popularity within the NLP community, in particular since (Rajpurkar et al., 2016) released a large-scale dataset (SQuAD) consisting of 100,000+ questions on a set of Wikipedia articles. In the medical domain, (Suster and Daelemans, 2018) created a dataset of clinical case reports for machine reading comprehension (CliCR). The dataset contains around 100,000 gap-filling queries on 12,000 case reports. These queries are created by blanking out medical entities in the learning points sections using some heuristics. We explore the QA framework for min/max age value extraction. Various neural network models have been proposed for question answering but these models trained on SQuAD or CliCR do not work well in our scenario because these datasets do not contain the queries targeting the specific min/max age values expressed in the text. Therefore, we leverage instead the massive structured clinical study records on ClinicalTrials.gov and create the training data for our min/max age value extraction component.

Non-factual Information Detection
There has been a significant amount of research in detecting speculative language in scientific research articles (Hyland, 1998;Light et al., 2004;Kilicoglu and Bergler, 2008;Medlock and Briscoe, 2007;Farkas et al., 2010;Morante and Sporleder, 2012). Our task requires to extract information from definite statements, therefore we use a list of speculation cues to filter out sentences where min/max age information are expressed speculatively.

Approach
We develop a pipeline to extract factual min/max age information from clinical trial studies. We divide the task in two steps: 1) finding sentences containing min/max age information; 2) extracting the value from those sentences. For the first we develop Min/Max age Sentence Classifier and for the second we propose a QA approach and develop the module Min/Max Age QA Model. Figure 1 illustrated the process associated with our proposed system. In the following sections, we describe how we create training data from ClinicalTrials.gov as well as each component of our system in detail.

Creating Training Data Using Clinical Study Records
We leverage the massive structured clinical study records on ClinicalTrials.gov to create training data for Min/Max Age Sentence classifier and Min/Max Age QA Model. ClinicalTrials.gov is one of the largest database of clinical studies conducted around the world. It currently holds registrations around 273,000 trials from 204 countries. Each trial registration record contains a column called "Eligibility Criteria", additionally min and/or max age values are indicated if they are present in the description text of the eligibility criteria. Figure 2 shows an example of a clinical study record from ClinicalTrials.gov. Note that most min/age expressions in eligibility criteria are speculative (e.g., at least 21 years of age, or child must be ages 6-12 years old), nevertheless they are still reflective of various linguistic forms for factual min/max age (e.g., aged 6-12 years old or age ≥ 18 years ). Therefore we expect that the models trained on this "noisy" dataset can still (1) identify sentences containing min/max age information and (2) predict the min/max age values.

Pre-processing
Given a research article in PDF format, we first extract clean text from the PDF file using GRO-BID (Lopez, 2009). We associate each paragraph to one of the five main sections: abstract, introduction, method, result and discussion. This step may introduce some noise (e.g., including the content from the table as the main body text) because parsing PDF file in different styles is a challenging task in itself.

Identifying Sentences Containing
Min/Max Age Information:Min/Max age Sentence Classifier After pre-processing, we identify sentences that contain min/max age information. At the inference stage, we first split paragraphs to sentences using Stanford CoreNLP Toolkit (Manning et al., 2014), then apply a classifier (Min-MaxAgeSentFinder) to predict sentences containing min/max age information among all sentences containing the word "age/ages/aged" or "year/years".
To train our MinMaxAgeSentFinder classifier, we create the training data using the eligibility criteria of the structured clinical study records from ClinicalTrials.gov. Text in eligibility criteria can be quite long (for instance, some criteria contain more than 10 clauses/sentences), so we only keep the clause/sentence which contains the annotated min/max age value(s). More specifically, we first split the eligibility criteria into sentences/clauses using the delimiter "-", then choose the clauses/sentences which contain the annotated min/max age values as well as the word "age/ages/aged" or "year/years". For instance, in the example shown in Figure 2, we will keep the sentence "Women and men at least 21 years of age with suspected NSCLC to be confirmed after surgery." as the positive training instance and filter out other sentences/clauses. We randomly choose 20,000 such sentences/clauses (10,000 for min age and 10,000 for max age) as positive training instances. Negative training instances are sentences which do not contain the word "age/ages/aged" or "year/years" from 60 clinical research articles. Note that these articles are different from the articles in the testing dataset. We use MaxEnt classifier to train MinMaxAgeSentFinder with the following features: adjacent word n-grams (n=1-4) and adjacent letter n-grams within words.

Predicting Min/Max Age Values for Each
Sentence: Min/Max Age QA Model We approach the problem of extracting values of min/max age from a question-answering perspective. Specifically, our system first reads a sentence, then answers the questions "what is the min/max age of the participants?". Various neural network models have been proposed for this task but these models trained on SQuAD do not work well in our scenario, because SQuAD does not contain this type of questionanswer pairs. Therefore we create training data for max/min age value extraction by leveraging the massive structured clinical study records from ClinicalTrials.gov The training data are 10,000 <eligibility criteria-min age> pairs and 10,000 <eligibility criteria-max age> pairs described previously. Note that we use the whole eligibility criteria instead of choosing the specific sen-tence/clause which contain the min/max age value. We believe that with the additional min/max age information, the question-answering module can locate the position of the min/max age value and learn various patterns for the target question.
We train our min/max age question-answering module (MinMaxAgeQA) using the Bi-Directional Attention Flow (BiDAF) Network (Seo et al., 2017). BiDAF uses attention mechanisms in both directions (i.e., question-to-context and contextto-question) to find a sub-phrase from the input text to answer the question. BiDAF includes both character-level and wordlevel embeddings. Most word tokenization models are not robust for numeric expressions in scientific literature. For instance, the Stanford CoreNLP tokenizer tokenizes the clause: "aged 6-12 years old" as "{aged, 6-12, years, old}" -it does not recognize 6 and 12 as two different tokens. The character-level embeddings in BiDAF can overcome this problem and the module correctly predicts 6 is the value of min age for this example.

Predicting Min/Max Age Values for Each Article
To predict min/max age values of the study samples for each article, we apply MinMaxAgeQA to each predicted sentence containing min/max age information from the abstract, method, and result sections on both questions (i.e., what is the min age of the participants? and what is the max age of the participants?). Answers that do not represent a valid integer number or answers whose confidence score are less than 0.5 are discarded. For each question, we keep the answer with the highest confidence score. We do not include sentences from the introduction section because it may include other min/max age information which is not related to the study samples (see Example 5 and Example 6). We leave filtering out unrelated min/max age information from introductions as future work.
Finally, if both min and max age values are predicted for an article, we check whether the min age value is smaller than the max age value. Otherwise we keep the answer with the higher confidence score and discard the other one. For instance, as shown in Figure 3, the number 16 is predicted as both the min age value (with the probability of 0.956) and the max age value (with the probability of 0.624) for an article, we keep 16 as the prediction for the min age value and set the prediction of the max age to "Null".

Non-factual Age Expression Filter
In this component, we filter out a min/max age value prediction if it is expressed speculatively. We first extract the clause which contains the prediction, then check whether a speculation cue word/phrase is present in the clause using the speculation cues from (Light et al., 2004). These cue words are: {if, at least, must, had to, has to, have to, need, needs}.

Testing Dataset
The ground-truth dataset used for evaluation comprises a set of 50 published journal articles in PDF format on smoking cessation. The dataset contains around 432k tokens and 18k sentences. Table  1 shows some statistics about the testing dataset. Overall, we have 843 sentences containing the word "age/ages/aged" or "year/years" and these sentences contain 2,226 numeric tokens.
The articles were annotated by a team of four behaviour science domain experts in the context of a broader project focused on leveraging the scientific literature in behaviour change (Michie et al., 2017). Annotation for a particular document was performed by two human annotators using the EPPI tool 1 . The annotation process involved highlighting relevant pieces of text and then assigning them to the corresponding min/max age attribute. Additionally, in order to disambiguate the highlighted text, the annotators were asked to annotate the entire sentence containing the highlighted piece as the additional context. Conflicts in the annotation process were resolved through discussions. Note that not every document contains a min/max age annotation. This is because not every article reports the min/max age of the overall Testing Dataset # of articles 50 # of sentences 18,417 # of tokens (main text) 432,056 # of sentences containing 843 "age/ages/aged" or "year/years" # of numeric tokens in sentences containing 2,226 "age/ages/aged" or "year/years"

Evaluation Metric
We use recall, precision and F-score for evaluation. Recall is calculated as the number of articles where the min/max age values are correctly predicted divided by the number of articles where min/max age values are annotated. Precision is calculated as the number of articles where the min/max age values are correctly predicted divided by the number of articles where the system makes a min/max age value prediction. F-score is the harmonic average of the precision and recall.

Baseline 1: PassageRetrievalBasedMin-MaxAgeExtractor
We developed a passage retrieval based IE system to extract min/max age values (Ganguly et al., 2018). The first step is to retrieve the passages containing 10, 20, and 30 words using the query "(age OR ages OR aged OR year OR years)". The intention of retrieving passages is to restrict extraction of factoid answers to potentially relevant small semantic units of text rather than the text of the whole document. The next step is to use validation criteria to select the likely answer candidates. We use the min/max age patterns from (Summerscales, 2013) as the validation criteria to choose the likely answer candidates from each retrieved passage for min age and max age respectively. These patterns can be viewed as rules which are carefully designed by humans to extract min/max age values. For instance, a rule can be: if a passage contains the phrase "greater than X" or "older than X" and X is an integer number between 10 to 100, then choose X as an answer candidate. It is worth noting that (Summerscales, 2013) is the only pre-vious work targeting the same task according to our best knowledge. We integrate all the heuristic rules for min/max age value extraction from (Summerscales, 2013) into our passage retrieval based IE system.
Finally, we score the answer candidates by a term proximity function that takes into account the differences in position between the query terms and the candidate answers (Zhao and Yun, 2009). The function is formally defined in the following Equation: Equation 1 describes the proximity based ranking function between a candidate answer c and a query Q, denoted by sim(c, Q). Practically, for each word in the passage that matches the query terms (q), the similarity function increases the score of that candidate by an amount that depends on the distance between that matched word and the candidate answer (p c − p q ). Specifically, we use a Gaussian function centered at each query term to determine the increase in similarity score. The parameter σ controls the bandwidth of the Gaussians and is set to 1 in our experiments.

Baseline 2: CRFBasedMinMaxAgeExtractor
We also developed the second baseline using CRF (Sutton and McCallum, 2012). The training dataset contains the clauses/sentences which contains the annotated min/max age value(s) from the eligibility criteria of the clinical studies registered in ClinicalTrials.gov. For each clause/sentence, we use Stanford CoreNLP Toolkit (Manning et al., 2014) to obtain the tokens as well as the POS tags, then we create the corresponding training instance using BIO labels (i.e., Beginning/Inside/Outside of a min/max age). Table 2 shows the training instance for the example illustrated in Figure 2.  We train two CRF models for min age and max age extraction respectively, using 10,000 training instances for each model. We use words as well as POS tags as features. More specifically, for the word type features, we consider the current word w i , the surrounding words (w i−2 , w i−1 , w i+1 , w i+2 ), as well as bi-grams (w i−1 + w i , w i + w i+1 ) and tri-grams (w i−2 + w i−1 + w i , w i−1 + w i + w i+1 , w i + w i+1 + w i+2 ) created from words. We create similar unigram, bi-gram and tri-gram features using the automatically predicted POS tags as well. We also include the combinations of the previous prediction and the current prediction as bi-gram features.
At the inference stage, for each article, we first extract all sentences containing the word "age/ages/aged" or "year/years". We then apply the min/max age CRF model on these sentences and extract all tokens with the predicted label "B". In the end, among all predicted words, we choose the word which represents a valid integer number and has the highest confidence score as the predicted min/max age value for the article. Table 3 shows the performance of the baselines (PassageRetrievalBasedMinMaxAgeExtractor and CRFBasedMinMaxAgeExtractor) as well as our system (QABasedMinMaxAgeExtractor, described in Section 3) for extracting min/max age values of the study samples.

Results and Discussion
For MinAge, the first baseline (PassageRe-trievalBasedMinMaxAgeExtractor) achieves a very high precision score (90.9%) but suffers from low recall (28.6%). The second baseline (CRFBasedMinMaxAgeExtractor) improves the recall by 21.4 points but only achieves a precision score of 42.5%. Compared to the first baseline, our system manages to improve recall by 37.1 points and still achieves a reasonable level of precision (79.3%). Overall, our system improves the results over the two baselines by a large margin regarding F-score (71.9% vs. 43.5%, and 71.9% vs. 45.9%).
The similar pattern is also observed for Max-Age: Our system improves the results over the first baseline by a substantial margin on recall (60.0% vs. 32.0%) and F-score (66.7% vs. 44.4%) respectively.
It might seem surprising that CRFBasedMin-MaxAgeExtractor performs much worse than PassageRetrievalBasedMinMaxAgeExtractor for MaxAge. This is because many max age values in scientific articles are not correctly recognized as a single token by Stanford CoreNLP Toolkit. For instance, the tokenization model predicts that "18-60" or "<=60" as single tokens. In contrast, our system is more robust for parsing such numeric expressions.
In addition, it seems that the carefully designed min/max age patterns in the first baseline only cover a few forms of min/max age expressions. On the contrary, our min/max age question-answering module (MinMaxAgeQA, Section 3.4) trained over a large-scale dataset can capture various linguistic expressions of min/max age in natural language, for instance, "≥ 18 years of age" or "age >= 18 years".

Analysis
To better understand the roles of different components in our system, we carried out a few experiments: • -WO MinMaxAgeSentFinder: instead of using MinMaxAgeSentFinder to find the sentences containing min/max age information, we pass all sentences containing the word "age/ages/aged" or "year/years" from the abstract, method, and result sections to the next component MinMaxAgeQA.
• -WO MinMaxAgeQA: we use the most common min/max age expression pattern in clini-   cal trial studies "X-Y" (e.g., 18-23 years old) to predict min and max age values from the first sentence contain such a pattern.
• -WO Non-factualSentFilter: Non-factual age expression filter is not used.
The results of these experiments are shown in Table 4. It seems that MinMaxAgeQA has the most impact on the performance while Non-factualSentFilter has less of an impact. In addition, MinMaxAgeSentFinder has more impact on the results of MaxAge compared to MinAge.
We also performed some error analysis on our full system. We noticed that the noise introduced in the pre-processing step (e.g., missing some paragraphs) is the main reason to cause our system to predict "Null" for articles with min/max age annotation. For cases where a wrong min/max age value is predicted, they are often embedded in the speculative expressions which are not captured by our current Non-factualSentFilter. For instance, the system predicts 24 as the max age for one article in which 24 appears in a speculative sentence (see speculative expression in Example 7). For this article, the annotation for max age is 23 (see factual expression in Example 7).
(7) (speculative expression) Eligibility for this study included being a student (full or part time), smoking at least 1 cigarette/day in each of the past 7 days, being aged 18-24 years, and being interested in quitting smoking in the next 6 months. (factual expression) Participants were 83 smok-ers, who were 18-23 years old and undergraduate students at a university.

Conclusions
This paper aims to extract factual min/max age values of the study samples from clinical research papers. We leverage the large-scale records from the ClinicalTrials.gov database to provide distant supervision for our system. We also explore "speculative cues" and the structure of the scientific papers to extract information from factual statements about the target study. We show that our approach outperforms a passage retrieval based IE system and a CRF-based model by a large margin on a testing dataset consisting of 50 journal articles and around 18,000 sentences.
In the future, we plan to extend our framework to extract other types of numeric values from the clinical research papers, such as the outcome values of the different intervention groups and the control group (e.g., 40% of PP abstinence rates), as well as the time frame of the follow up (e.g., 52 weeks or 6 months).


Image Shadow Removal Using End-To-End Deep Convolutional Neural Networks

Image degradation caused by shadows is likely to cause technological issues in image segmentation and target recognition. In view of the existing shadow removal methods, there are problems such as small and trivial shadow processing, the scarcity of end-to-end automatic methods, the neglecting of light, and high-level semantic information such as materials. An end-to-end deep convolutional neural network is proposed to further improve the image shadow removal effect. The network mainly consists of two network models, an encoder–decoder network and a small refinement network. The former predicts the alpha shadow scale factor, and the latter refines to obtain sharper edge information. In addition, a new image database (remove shadow database, RSDB) is constructed; and qualitative and quantitative evaluations are made on databases such as UIUC, UCF and newly-created databases (RSDB) with various real images. Using the peak signal-to-noise ratio (PSNR) and the structural similarity (SSIM) for quantitative analysis, the algorithm has a big improvement on the PSNR and the SSIM as opposed to other methods. In terms of qualitative comparisons, the network shadow has a clearer and shadow-free image that is consistent with the original image color and texture, and the detail processing effect is much better. The experimental results show that the proposed algorithm is superior to other algorithms, and it is more robust in subjective vision and objective quantization.

Introduction
Today's world is an era of rapid information development. With the continuous upgrading of multimedia technology and electronic products, human beings are exposed to more multimedia information such as text, images, and audio in their daily life. Most of this multimedia information is obtained through people's visual systems. We define the multimedia information obtained through this channel as an image. However, when the image is acquired, it is susceptible to various conditions that generally degrade the quality of image. Shadows are one of them. Shadows are a phenomenon of quality degradation caused by imaging conditions. These conditions lead to missing or interfered information reflected by the target; therefore, the interpretation accuracy of the image is sharply reduced, and it affects various quantitative analyses and applications of the image.
Shadow detection and removal [1][2][3][4] is one of the most basic and challenging problems in computer graphics and computer vision. The removal of shadow images are important pre-processing stages in computer vision and image enhancement. The existence of shadow not only affects the visual interpretation of the image, but also affects the analysis of the image and the subsequent processing results. For instance, a darker area (caused by shadows) introduces incorrect segments in image

Related Work
In recent years, many scholars have analyzed the characteristics of shadows, established shadow generation models, and proposed a number of algorithms for shadow detection and shadow removal. Analyzing the research results comprehensively, it is not difficult to find that the shadow removal algorithm mostly follows these principles: detecting the shadow or marking it manually, then creating models to remove the shadow. Gong and Cosker [15] proposed a statistical-based, interactive shadow removal method for shadows in extreme conditions with different shadow softness, irregular shadow shapes, and existing dark textures. First, using the dynamic learning method, two users roughly marked the shadow and non-shadow areas in the image, meaning the shadow area was highlighted and the non-shadow area was resisted, so the shadow area could be extracted easily. On this basis, the model was constructed for shadow removal. Gong and Cosker used a rough hand-marked form to detect shadows while sacrificing finer shadows and a full range of simpler user input autonomy. Gryka [16] and Yu [21] et al. also proposed to manually mark the shadows in an interactive manner to achieve the purpose of shadow removal. Different from Gong and Cosker, Yu [21] proposed a color line-based artificial interactive shadow removal algorithm, which required users to provide shaded areas, shadow-free areas, and areas with similar textures changing significantly in brightness. In the work of Gryka [16], they used the unsupervised regression algorithm to remove the shadows through inputting the shaded area as processed by user, and mainly focused on the soft shadows in the real scene.
Finlayson [22] proposed a shadow removal algorithm based on Poisson's equation, which had a good effect on a considerable part of the image. However, there was no consideration of the influence of ambient illumination and material changes, thus resulting in poor texture restoration in shadow areas. Generally, the brightness contrast between the shadow area and the non-shadow area in the image are relatively large while the features such as texture and color are similar. Therefore, the shadow area and the non-shadow area have similar gradient fields. Liu et al. [23] used this feature to propose an image shadow removal algorithm based on gradient domain, which solved some shortcomings of Poisson equation, but it had a poor performance on discontinuous or small shadow regions. Xiao et al. [24] proposed a shadow removal algorithm based on Subregion Matching Illumination Transfer. This method takes into account the different reflectivity characteristics of different materials, which makes the processing of shadow regions in complex scenes better.
On the one hand, Zheng [25] proposed a projection shadow detection and elimination algorithm based on the combination of texture features and luminance chrominance chroma (YUV) color space. Firstly, a moving object was detected using a pixel-based adaptive segmenter (PBAS) algorithm that resisted shadows. In addition, a portion of the projected shadow candidate area was obtained by texture detection. Then, another projected shadow candidate area was obtained by shadow detection that was based on the luminance chrominance chroma (YUV) color space. Finally, the two portions of the projected shadow candidate regions were filtered and merged by the shadow features. In the work of Murali [26], they proposed a simple method for detecting and removing shadows from a single red green blue (RGB) image. The shadow detection method was selected based on the average value of the RGB images in the A and B planes of the CIELab color model (LAB) equivalent image, and the shadow removal method was based on the recognition of the amount of light irradiated on the surface. The brightness of the shaded areas in the image was increased, and then the color of the surface portion was corrected to match the bright portion of the surface. Tfy et al. [27] trained a kernel least-squares support vector machine (LSSVM) to separate shadow and non-shadow regions. It was embedded it into the Markov random field (MRF) framework and pairwise contextual cues were added to further enhance the performance of the region classifier so the detection of shadow area was realized. A shadow removal method based on area re-lighting is raised based on this.
On the other hand, for the umbra and penumbra regions in projection, researchers have proposed an image shadow removal algorithm based on convolutional neural network structure (CNN) [14,18] or intensity domain [9,11] in deep learning. Both Khan [14] and Shen [18] used convolutional neural networks (CNN) for learning. The former used multiple CNN structures for fusion applications and conditional random field (CRF) for image shadow region predictions, while the latter applied structured label information to the classification process. The two methods were different in the process of extracting and removing shadows. The former used the Bayesian framework, while the latter applied the least-squares method of optimization to perform the final recovery optimization. Arbel et al. [9] used the Markov random field to determine the penumbra region, and then created a smoothing model in the shadow region to construct a shadow mask. Wu [11] extracted the shadow mask by considering features such as color shift, texture gradation, and shadow smoothness. Meanwhile, it reserved the texture of the shadow area during the extraction process.
Analyzing and summarizing the above methods, we find that the current method of shadow removal can either restore the texture of the shadow area effectively, or neglect the influence of environment and material of the object. Most methods are interactive rather than fully automated, which greatly reduces the efficiency of use. Therefore, the purpose of this paper is to propose an end-to-end, fully automatic network structure to remove image shadow.
With the development of artificial intelligence in recent years, some researchers have established fully automatic models for image shadow removal [28][29][30][31]. A shadow removal algorithm based on generative adversarial networks (GANs) is proposed in the work [30], which uses two GANs for joint learning in order to detect and remove shadows. Although this allowed fully automatic shadow detection using GANs, it relied on the shadow detection result of the previous generation against the network in the shadow removal stage, and it did not perform well when dealing with small shadow areas and shadow edges. In the work of Yang [28], a three-dimensional inherent image restoration method based on bilateral filtering and a two-dimensional intrinsic image is proposed. Two-dimensional original images are derived from the original image, and then the shadow restoration method of the three-dimensional original image is proposed by deducing two-dimensional original images and bilateral filtering. Finally, an automatic shadow removal method for a single image is realized. However, since the reconstruction of the image may result in changes in the non-shaded area, a high-quality image consistent with the shadow-free image generally cannot obtained.
Therefore, this paper aims to explore the end-to-end deep convolutional neural network for image shadow removal. An encoding-decoding neural network is used, and a small neural network with a two-layer neural network structure is used to train learning. Then, the original image (shadow image) is inputted into the trained network structure to realize fully automatic shadow removal. Finally, a high-quality, shadow-free image is obtained.

End-To-End Convolutional Neural Network
This article aims to solve the shadow removal problem using an end-to-end deep convolutional network structure, which is named RSnet. It aims to learn a mapping function between the shadow image and its shadow matte. The RSnet model mainly includes two parts: an encoding-decoding stage and an elaboration stage. Specifically, we used a network structure of a deep convolutional encoder-decoder that took an image as the input, and it took into account penalties for the shadow mask factor (α) that predicted loss and penalties for new composition loss. At the same time, a refined, small network structure was introduced in order to consider multi-context scenarios as well as process local information and edge information, etc. The related information of the two networks, the training of RSnet model, and how to get the shadow-free image will be described in detail in the following chapters.

Encoding-Decoding Phase
The deep encoding-decoding network structure is a very common model framework in deep learning. The encoder and decoder parts can be any text, voice, image, or video data. The model can use CNN, recurrent neural network (RNN), long short-term memory (LSTM), gated recurrent unit (GRU), and so on. One of the most notable features of the encoder-decoder framework is that it is an end-to-end learning algorithm. It has been widely used in various fields and has succeeded in many other computer vision tasks, such as image segmentation [32], object edge detection [33], and image restoration [34]. The encoding-decoding network structure in this paper consisted of two parts, one was the encoder network for extracting features and the other was the decoder network for reconstructing images. Therefore, the encoder network inputted a shadow image, and the output was a related feature; the decoder network took the output of the encoder network as an input and finally outputted a shadow mask through the deconvolution layer and the unpooling layer.
Encoder network: The encoder network was based on the VGG16 network [35], which was originally designed for object recognition. Recent studies have shown that CNNs train a large amount of data on image classification tasks and can be well generalized in databases and tasks, such as depth prediction [36,37] and semantic segmentation [38]. Therefore, this paper used the pre-trained convolution layer of the VGG16 model [35] and transferred its feature representations to shadow masks by fine-tuning, for the function of task-prediction.
As is known, the VGG16 network consists of thirteen 3 × 3 convolutional layers (five convolutional blocks) and three fully connected layers, as well as five maximum pooling and subsampling layers. These five convolutional layers and spatial aggregations greatly increased the acceptance of the network, so that the characteristics of the global context and semantic scenes could be extracted. However, these five largest pooling layers introduced a 32-pixel stride in the network, so the final prediction graph was quite rough. Therefore, in order to extract more detailed features and obtain more intensive predictions, the step sizes of the first, third, and fifth layer pooling layers in the VGG16 model were modified to 1. In addition to this modification, the sub-sampling layer was abandoned, and all fully connected layers in the VGG16 network were replaced with a 1 × 1 convolutional layer [38]. These 1 × 1 convolutional layers enabled the network to operate in a fully convolutional manner. Specifically, the encoder network had a total of 14 convolutional layers and five largest pooling layers.
Decoder network: In the decoder stage, this paper used the deconvolution network proposed by Zeiler [39], which was mirror-symmetric with the convolutional network. The deconvolution network had an unpooling layer and deconvolution layer. With the successful application of deconvolution in neural network visualization, it had been adopted in more and more work, such as scene segmentation, model generation, etc. Deconvolution is also known as transposed convolution, and it can be clearly seen that the forward propagation process of the convolutional layer is the reverse propagation process of the deconvolution layer, and the reverse propagation process of the convolutional layer is the forward propagation process of the deconvolution layer. The unpooling layer captures instance image space information. Therefore, it can effectively reconstruct the detailed structure of an object with finer resolution.
However, in order to speed up the training of the network, the network structure of the VGG16 was rejected. A small network structure was defined to sample the features acquired in the encoding network to obtain the desired output. Specifically, the decoder network was defined as eight convolutional layers, five deconvolution layers, and the last layer was the prediction layer of the shadow mask factor α.
In order to prevent over-fitting and achieve local optimum values, dropout was applied after each convolutional layer of the encoding-decoding network, so that the over-fitting problem could be effectively mitigated and achieve the regularization effect to some extent. This article no longer used the activation function rectified linear unit mentioned in the VGG16 network, ReLU [40]. Instead, the parameter rectification linear unit (PReLU) was used throughout the encoding-decoding phase, which was an improved ReLU proposed by the He Kaiming group [41] and featured non-saturation. PReLU introduced an additional parameter to avoid the over-fitting problems. The definition of PReLU is as follows: where x i is the input to the activation function p at channel i, and λ is the learning parameter.
In particular, when λ is a fixed non-zero smaller number, it is equivalent to LeakyReLU (a special version of the Rectified Linear Unit (ReLU)); when the parameter λ is 0, PReLU is equivalent to ReLU.

Refinement Phase
While the first part of the RSnet network prediction worked well, the results could sometimes be too smooth as a result of the encoder-decoder structure. Therefore, we designed a multi-context mechanism for local detail correction and the previous stage of the encoding-decoding network. Thereby, the results of the first phase were further improved so that the prediction result α of the entire network structure was more precise, and the edge processing effect was more refined.
Network structure: The refinement phase was the second phase of the RSnet network, which aimed to improve the prediction result α obtained in the first phase. Therefore, the input of this stage connected the image patch with the obtained α predicted value of the first stage and produced a 4-channel input, and the output was the true shadow scale factor α in the image. To ensure convergence speed and granularity, the network was designed as a small, fully convolutional network with three convolutional layers. Each one had a nonlinear ReLU layer without a down-sampling layer in the first two convolutional layers, as the purpose of this article was to preserve the very subtle structures in the first phase. The specific configuration is shown in Figure 1. In the refinement phase, instead of making large-scale changes to the alpha mask, we refined and sharpened solely the alpha values.

Training
The relationship between the shadow image I s and its shadow mask α is given by Equation (1). In training, the actual shadow mask α was calculated according to a given pair of shadows and shadow-free images. Then, to establish the relationship between the shadow image I s and the shadow mask α, the mapping function was as follows: Among them, β was the learning parameter of the convolution network. During training, a linear fusion of two loss functions was used to adaptively train the RSnet model. α prediction loss: The first loss was called α prediction loss, which was the difference between the absolute ground actual α value and the predicted α value of each pixel. The mean square error (MSE) is used as a loss function to approximate it: where N is the total number of training samples in the batch, F(I i s , β) is the shadow mask of the ith image pair output by the prediction layer through the training network, and α i is the true shadow mask of the ith image pair.
Composition loss: The second loss was called the composition loss, which was the discrepancy between the true shadow image RGB color and the predicted shadow image RGB color. The predicted shadow image RGB color was synthesized by the shadow-free image and the shadow layer factor α outputted in the prediction layer. The loss function approximation is as follows: where c represents the RGB channel of the real shadow image, c represents the RGB channel of the predicted shadow image synthesized by the predicted shadow mask factor α , ε represents a small value, and 10 −5 is taken in this training. The derivative ∂L ∂c is easy to calculate, its formula is defined as follows: ∂L ∂c Overall loss: It is a linear fusion of two losses, namely: Here, λ was set to 0.5. Training method: Although the performance increased significantly with the increase of network depth, it was an extraordinary task to train a deeper network due to the problem of vanishing gradient and exploding gradient. In this paper, the following three methods were used to achieve fast convergence and avoid over-fitting: (1) Staged training. Since the RSnet network structure was a combination of two networks, each network was trained separately in the training process. When the network reached a certain precision, the two were combined and trained together, and finally the combined optimization of the two was realized. (2) Multi-level training. According to the size of the shadow scale factor, different levels of images were set for training, for instance, training hard shadow image databases before soft shadow image databases, and combining the two to form a database for training in the last step. At the same time, considering the difference in pixel size of the user input image, images of different pixel sizes can also be divided into multiple levels for training. All of those methods are more robust in this paper, and help the network to learn context and semantics information better, thus, multi-level training results were finally achieved. (3) Data augmentation. Hundreds of individual scene images cannot achieve good results for the purpose of training a deeper network. In order to diversify the training data and expand the number of data pairs in the shadow database, the training of multiple scenes images was carried out, and the database of the same scene was expanded by adopting the methods of cutting and rotating. All this had realized the diversification of the database.
Obtain shadow-free image: The RSnet model aimed to learn a mapping function between the shadow image and its shadow mask. When the RSnet model was trained to obtain the mapping function between the two, the shadow mask corresponding to the shadow image could be obtained through the trained RSnet model, and the corresponding shadow-free image could also be obtained according to the Formula (1).
In the Formula (1), I s is the shadow image, I s f is the shadow-free image, α is the shadow mask, and ⊗ is the product of the pixel level.
To be specific, the shadow image was given by the user input, and the shadow mask was obtained by the trained RSnet model. The shadow-free image can be obtained by dividing them at the pixel level according to Formula (1).

Experiment
Given the complexity of the image composition, it was rather difficult to obtain accurate evaluations through visual observation alone. Therefore, this paper used RSV to evaluate RSnet on different databases. Specifically, during the experiment course, the UCF shadow database [42], the LRSS database [16], the UIUC shadow database [13], and the RSDB database (a new remove shadow database) were used for testing, respectively, through the RSnet network. The restored image and the shadow-free image were compared to ensure the diversity and enrichment of the data. At the same time, structural similarity (SSIM) and peak signal-to-noise ratio (PSNR) were used as evaluation indicators to evaluate the performance of RSnet.

Database
Shadow removal has important research significance. Researchers at home and abroad have been extensively studying this issue, especially in the past decade, but a database specifically constructed for this problem is very rare. Currently, the most widely used databases are presented by Guo [13] and Gryka [16], but they only contain 76 and 37 pairs of shadow/shadow-free image pairs. In order to better evaluate the shadow removal algorithm proposed in this paper, we constructed a new RSDB database, which contained 2685 shadow and shadow-free image pairs. The comparison with other shadow removal databases in recent years is shown in Table 1. In order to better collect data and build a new shadow removal database, we used a tripod to fix the camera in the specified position and a wireless Bluetooth remote control unit to capture images during the shooting process to ensure that the shadow/shadow-free image was consistent with the background. Firstly, to set a fixed exposure compensation, focal length, and other parameters, the exposure compensation was set to 0, and the focal length was 4 mm; then, the shadow image projected by different objects was taken. Finally, the shadow cast by the object was removed, that is, the projection of the object was removed, and the background image of the corresponding shadow image was captured, that is, the image without shadow.
In addition, to diversify the database, the new RSDB database had the following characteristics: Brightness: In order to capture shadows, databases with different softness, that is, shadow databases with different light intensities, included soft shadows and hard shadows. We collected shadow images at the same time in different weather conditions and different time periods in the same conditions. For example, we can take shadow pictures at the same time (including 11 noon) on cloudy and sunny days, or we can choose 7:00 in the morning, 12 in the afternoon, and 6 in the afternoon at different times of the day. The data was collected to obtain a soft shadow and hard shadow database.
Shape: In order to ensure the shape diversity of the shadow database, during the shooting process the shadows were projected by various shapes of objects such as school bags, umbrellas, bicycles, etc. In the meantime, the characteristics of subjective initiative are fully utilized, and the body was projected as a shadow. The object performed a multi-pose change and then projected it into a specified area, thus obtaining a shadow database with various shapes.
Scenario: Multiple scenes were another requirement for shadow removal databases. Given this, we collected separate shadow images for different scenes in the process of shadow-collecting, such as grass, roads, campus, and walls.
Reflection: When we were shooting a shadow database, we reflect the shadows of different objects to mirror the existence of shadows in real life more realistically. An example of an RSDB database is shown in Figure 2.  (a2, b2, c2) columns indicate true shadow-free images (targets) corresponding to the (a1, b1, c1) columns.

Performance Comparison Analysis
In order to evaluate the performance of the RSnet network, this paper chose to use the structural similarity SSIM (structural similarity) and the peak signal-to-noise ratio (PSNR) as evaluation indicators to assess the model. On this basis, the RSnet algorithm proposed in this paper was compared with several popular algorithms in recent years.
SSIM is an index to measure the similarity between two images. It is widely used to evaluate the quality of image processing algorithms. The larger the SSIM value of two images, the higher the similarity between them and the better the corresponding shadow removal effect in the model is. Conversely, the smaller the value of SSIM, the lower the similarity between the two, and the worse the removal effect of the corresponding algorithm is. PSNR is the most common and widely used objective measurement method for evaluating image quality. It is similar to SSIM. When the PSNR value of two images is larger, it indicates that the corresponding shadow removal model is better. Otherwise, the smaller the PSNR value, the worse the removal effect of the corresponding algorithm is. (Notably, the PSNR score is not exactly the same as the visual quality seen by the human eye. It is possible that a higher PSNR seems to be worse than a lower PSNR. It is because the sensitivity of the human eye to the error is not absolute, and the perceived result is affected by many factors.) The definitions of SSIM and PSNR are as follows: It is worth noting that in order to better reflect the removal effect of the shadow image, we chose shadow/output and target/output as the quality measurement properties of the algorithm, where shadow/output represented the corresponding evaluation index value between the original shadow image and the shadow removal obtained by the algorithm. target/output represented the actual ground shadow-free image (target) and the evaluation index value corresponding to the shadow removal image obtained by the algorithm. In order to evaluate the performance of the algorithm more fairly, we only showed the shadow/output values of the UCF database in different algorithms, and the target/output value is not given since the UCF database did not provide a true ground-free shadow (target).
It can be clearly seen from Table 2 that the SSIM index values obtained in the RSnet model were superior to the SSIM values of other algorithms, regardless of shadow/output or target/output. In particular, as can be seen from the last line of Table 2, in the newly created RSDB database, when the SSIM values of other algorithms were compared with the SSIM values obtained by the RSnet model of the algorithm, the SSIM value of the former was lower, the shadow removal effect of the image was poorer, and the value of the RSnet model was much higher than other algorithms. Particularly, in the evaluation of the real ground shadow-free image and the shadow removal image obtained by the algorithm, it can be seen from Table 2 that it was 1.09 times the SSIM value obtained by the shadow removal algorithm proposed by Yu [21], and 1.47 times the SSIM value obtained by the model proposed by Yang [28].
Of course, the comparison on the database proposed in this paper was not standardized. Therefore, Table 2 also shows the SSIM values on other shaded databases such as LRSS. By analyzing the SSIM values on other databases in Table 2, it can be found that RSnet exhibited better robustness and better image shadow removal. The processing effect was much better, especially in the third line UIUC database, which demonstrated a desired shadow removal effect. The SSIM value between the original shadow image and the algorithm-derived shadow removal image was larger than the value under the LRSS database. The overall effect of the SSIM value was smoother and more stable than other databases. From the perspective of the algorithm, the algorithms proposed by Wang [30] and Qu [31] were both based on deep learning, and both of them had better performances compared with the other five algorithms, indicating that the shadow removal effect of them was better than other algorithms. Even for UIUC data sets, the SSIM value between the shadow removal image (output) and the real ground shadow-free image (target) obtained by the model proposed by Wang [30] was the best, which was even better than the model proposed in this paper. It can be concluded that the model proposed by Yang [28] had the worst overall performance, since it was easy to change the non-shaded area in the original image during the shadow removal process, which resulted in poor similarity between the two. The image shadow removal model proposed by Yu [21] and Gong [15] was relatively better than other algorithms, but it was inferior to the RSnet model proposed in this paper. The RSnet model had more stability, and the image shadow processing effect was better. The PSNR value in Table 3 showed a large difference between the shadow/output and target/output values when comparing the shadow/output (target/output) values of different algorithms under the same database or different databases of the same algorithm. We uncovered some valuable clues, which are needed to evaluate the algorithm. Comparing the PSNR values of different algorithms in the same shaded database in Table 3, it can be found that the value of the RSnet model was higher than other algorithms, especially in the RSDB database proposed in this paper, and the PSNR value was higher. The effect of the shadow removal model proposed by Yu [21] was similar to that of Gong [15]; the model PSNR values proposed by Yang [28] were relatively inferior, and so was the shadow removal performance. Qu [31] proposed an algorithm that had good performance, second only to the model proposed in this paper. As can be seen from the fourth line in Table 3, Qu's [31] proposed algorithm was superior to the model proposed in this paper, and the PSNR value of the model proposed by Wang [30] was slightly lower than that of the RSnet model. The three methods based on deep learning were all superior to the other five algorithms, which showed that the deep learning method achieved good results in the image shadow removal task. Comparing the PSNR values of the same algorithm under different databases, it can be clearly found that the proposed algorithm was different from other algorithms, and the PSNR values were higher under different databases. Therefore, the RSnet model was more stable, the image shadow removal effect was much better, and the algorithm was more robust. Comparing the PSNR values of the shadow removal model proposed by Guo [13] under a certain attribute and different databases, for example, the PSNR value on the shadow/output can be found, and the variation range of the PSNR was larger, indicating that the corresponding algorithm was not stable and the robustness was poor. At the same time, it can be concluded from Table 3 that the model proposed by Finlayson [22] had a smaller PSNR value under the target/output attribute of the RSDB database. Despite that it was higher than the model proposed by Yang [28], it was still 1.27 times lower than the value obtained by RSnet proposed in this paper. Table 3. Quantitative results using peak signal-to-noise ratio (PSNR) (the bigger the better), the maximum value is shown in bold.

Qualitative Analysis
In order to demonstrate the effect of RSnet network shadow removal properly, we selected part of the image in the database as the original image, and compared the processing results with the shadow removal effects of several other algorithms. The comparison diagram is shown in Figures 3-5.   5 show the image shadow removal effects of different algorithms under simple semantics, complex semantics, and special semantics. The first column is the original image (shadow image), the second column is the real shadow-free image, and the last eight columns are the shadow removal images obtained by eight different algorithms. The last column is the shadow removal image obtained by the RSnet model proposed in this paper. It is worth noting that the first few lines of Figures 3 and 4 show the image shadow removal effect when the projection object is smaller and more trivial, and the last few lines show the shadow removal effect when the projection object is larger. Comparing the processing effects of these two different projection objects, it can be clearly found that when the projection object was larger and the shadow area was more concentrated, that is, the shadow occupied a large proportion of the entire image, the overall removal effect of the image shadow was much better than that of the trivial projection object, and had a smaller shadow area. Analyzing the image shadow removal effect diagram in Figure 3, we found that Finlayson [22] and Yu [21] had relatively stable results for shadow images in simple scenes; the former was better than the latter. The algorithm proposed by Yu [21] had a better overall effect than the other four methods without deep learning, however, when comparing the shadow removal effect map obtained by the fully automatic RSnet model proposed in this paper, it can be seen that Rsnet's shadow-free image quality was higher, especially for the detail processing, which was far better than the interactive method proposed by Yu [21]. Observing the last three columns in Figures 3-5, it can be found that the depth learning-based methods proposed by Wang [30] and Qu [31] and the Rsnet algorithm proposed in this paper were better than the previous ones. It reflected that the overall effect of deep learning applied to image shadow removal was better. It can be clearly seen from the fifth column in Figures 3-5 that the bilateral filtering-based algorithm proposed by Yang [28] changed the non-shaded area. Of course, for the first and second rows of the fifth column in Figure 3, the image reconstruction effect was good, the reconstruction effect of the non-shadow region was good, and the non-shadow region was not significantly changed. However, the effect of shadow removal was not good enough, and the shadow area could still be observed more clearly from the shadow-free image after removal. At the same time, using these two images to compare the shadow-free image obtained by the Rsnet model proposed in this paper, it can be seen that the latter removal effect was better in both the non-shadow area and the shadow area, and the obtained shadow-free image quality was higher. Of course, for the shaded areas in the third and last lines, the algorithm proposed in this paper did not perform preferably on the removal of shadow areas, but still had advantage over other algorithms, especially for the detailed processing of edge information.   Figures 3 and 4 shows that some algorithms had better shadow removal effects in simple scenes, but the shadow removal effect in complex scenes was relatively poor. These algorithms of the overall analysis and comparison show that the quality of the shadow-free image obtained in the simple scene was higher than that of the shadow-free image obtained in the complex scene. Therefore, it can be explained that these algorithms had poor stability and could not adapt to shadow images with multiple semantic inputs at the same time. When the texture was complex and the shadows were small, these algorithms could not obtain high-quality, shadow-free images. Of course, the two methods based on deep learning were more stable than the other five algorithms. However, when comparing the RSnet model, it can be found that Wang [30] had a poor effect on small shadows, and Qu [31] had a poor shadow removal effect on complex scenes, such as grass. The Rsnet model had excellent shadow removal effects for both scenarios, and both had high quality, shadow-free images. Analysis of lines 4 and 5 in Figure 4 revealed that when shadows were projected onto a wall with complex textures, Yu [21], Gong [15], Guo [13], Yang [28], and Finlayson [22] did not get high-quality, shadow-free images, and the shadow removal effect was not ideal. The shadow-free image quality obtained by the Rsnet network in this case was relatively higher, and the shadow removal effect was better, even equivalent to the true shadow-free image in the second column.
The last line in Figure 4 shows the shadow removal effect of different algorithms when people and bicycles were used as projection objects. This was a common shadow image in our daily life, but the shadow removal methods of the other models were not satisfactory. In particular, the more complicated and fine shadow processing effects on the bicycle tire was extremely poor, and the corresponding shadow area was not substantially removed. The Rsnet model proposed in this paper obtained a high-quality, shadow-free image using a fully automated end-to-end convolutional neural network. Moreover, when the obtained shadow-free image was compared with the real shadow-free image, there was basically no difference between the two, and the shadow removal effect was excellent. Figure 5 illustrates the effect of image shadow removal in some special cases. From the first three lines, it can be found that when the shadow area involved multiple semantic inputs, the other five shadow removal effects were extremely inferior. The model removal effect proposed in this paper was quite satisfactory. In particular, in the second row of Figure 5, which had stairs and involved two different colors, the Yang [28], Gong [15], and Guo [13] algorithms basically could not handle the shadow area, and they even changed the non-shadow area. From the last line in Figure 5, it can be found that when the real ground had a black texture, most of the shadow removal algorithms would mistake it for the shadow area to be removed, and the methods proposed by Wang [30] and Qu [31] presented the same situation. However, the RSnet model proposed in this paper could distinguish between shadow regions and non-shadow regions more accurately, and, thus, obtain higher quality, shadow-free images.

Summary and Outlook
Based on the analysis of the background, significance, and current research status of image shadow removal, this paper focuses on the shortcomings of existing image shadow removal algorithms. A fully automatic image shadow removal model based on an end-to-end deep convolutional neural network is proposed. This model uses a small-scale network to refine the processing that is based on the encode-decoder network to obtain finer semantic information, which enhances the quality of shadow-free images and better process the details. In the RSnet model, a more diverse training method is adopted in the training process, which performs the network training effect better and, thus, is applied to image shadow removal in a finer way. At the same time, a new image shadow removal RSDB database is provided, which offers more data types for future shadow removal. It can be seen from the comparison results that the image shading is removed by using this algorithm, the performance index is better, and the quality of the shadow-free image is higher, which has great application values in the later target recognition and target tracking stages. Of course, there are still some limitations with our method. In the following work, we would continue to optimize this method. The database size of the image shadow can be further expanded; the network expression ability can be improved, the convergence speed can be accelerated, and, thus, a more robust algorithm may be obtained.