extfieldsofstudy:
Computer Science
Medicine

text:
Automated deep learning design for medical image classification by health-care professionals with no coding experience: a feasibility study

performing deep learning algorithms. The performance in the external validation study was low. The quality of the open-access datasets (including insufficient information about patient flow and demographics) and the absence of measurement for precision, such as confidence intervals, constituted the major limitations of this study. The availability of automated deep learning platforms provide an opportunity for the medical community to enhance their understanding in model development and evaluation. Although the derivation of classification models without requiring a deep understanding of the mathematical, statistical, and programming principles is attractive, comparable performance to expertly designed models is limited to more elementary classification tasks. Furthermore, care should be placed in adhering to ethical principles when using these automated models to avoid discrimination and causing harm. Future studies should compare several application programming interfaces on thoroughly curated datasets.

Introduction
Diagnosis, an understanding of the probability for presence of illness, depends on data: its collection, integration, and interpretation enables accurate classification of clinical presentations into an accepted disease category. Human diagnosticians achieve acceptable accuracy in such classification tasks through the learning of diagnostic rules (patterns recorded by other human diagnosticians) followed by training on real cases for which the diagnostic labels are provided (supervised clinical experience). In artificial intelligence (AI), the technique of deep learning uses artificial neural networks-so-called because of their superficial resemblance to biological neural networks-as a computational model to discover intricate structure and patterns in large, high-dimensional datasets such as medical e233 www.thelancet.com/digital-health Vol 1 September 2019 images. 1 A key feature of these networks is their ability to fine-tune on the basis of experience, allowing them to adapt to their inputs, thus becoming capable of evolving. This characteristic makes them powerful tools for pattern recognition, classification, and prediction. In addition, the features discovered are not predetermined by human engineers, but rather by the patterns they have learned from input data. 2 Although first espoused in the 1980s, deep learning has come to prominence in the past 10 years, driven in large part by the power of graphics processing units originally developed for video gaming, and the increasing availability of large datasets. 3 Since 2012, deep learning has brought profound changes to the technology industry, with discoveries in areas as diverse as computer vision, image caption, speech recognition, natural language translation, robotics, and even self-driving cars. [4][5][6][7][8][9] In 2015, Scientific American listed deep learning as one of their "world changing ideas" of the year. 10 Until now, the development and implementation of deep learning methodology into health care has faced three main blockers. First, access to large, well curated, and well labelled datasets is a major challenge. Although numerous institutions around the world have access to large clinical datasets, far fewer have them in a computationally tractable form and with reliable clinical labels for learning tasks. Second, highly specialised computing resources are needed, because the performance of deep learning models depend on recent advances in parallel computing architectures, termed graphic processing units. The architecture of silicon customised to these tasks is rapidly evolving with software companies increasingly designing their own hardware chips, such as tensor processing units, and field-programmable gate arrays. 11,12 Thus, it is already clear that it will be difficult for small research groups, working alone in hospital and university settings, to accommodate these financial costs and the rapidly evolving landscape. Third, specific technical expertise and mathematical knowledge is required to develop deep learning models. This proficiency is still uncommon. A 2019 report 13 by Element AI concluded that although the number of self-reported AI experts worldwide had increased to 36 000, the "supply of top-tier AI talent does not meet the demand".
One approach to combat these obstacles is the increasingly popular technique called transfer learning, where a model developed for a specific task is repurposed and leveraged as a starting point for training on a novel task. Although transfer learning mitigates some of the substantial computing resources required in designing a bespoke model from inception, it nevertheless demands deep learning expertise to deliver effective results. With this in mind, several companies released application

Research in context
Evidence before this study We did a systematic search of the literature to identify classical deep learning models (with bespoke architectures developed by human experts) to provide a comparison to models constructed using the automated deep learning platform Google Cloud AutoML. We searched MEDLINE, Embase, Science Citation Index, Conference Proceedings Citation Index, Google Scholar, and arXiv from Jan 1, 2012, until Oct 5, 2018. We prespecified a cutoff of 2012, on the basis of a generally recognised step-change in deep learning performance since that time. No language restrictions were applied. Studies were included if the authors developed a deep learning algorithm on the datasets used in their study. The search strategy is available in the appendix. The performance of these existing models served as a direct comparison to the model we developed.
Our search resulted in five open-source datasets: retinal fundus images (MESSIDOR), optical coherence tomography images (Guangzhou Medical University and Shiley Eye Institute, version 3), images of skin lesions (Human Against Machine HAM10000), paediatric chest x-ray images (Guangzhou Medical University and Shiley Eye Institute, version 3), and adult chest x-ray images (NIH CXR14 dataset).

Added value of this study
Currently, the highly specialised technical expertise necessary to develop artificial intelligence (AI) models is scarce. Even transfer learning, which builds on existing algorithms, requires substantial machine learning experience to achieve adequate results on new image classification tasks. This currently limits the use of deep learning to a growing, but small, community of computer scientists and engineers.
We show, to our knowledge, a first of its kind automated design and implementation of deep learning models for health-care application by non-AI experts, namely physicians. Although comparable performance to expert-tuned medical image classification algorithms was obtained in internal validations of binary and multiple classification tasks, more complex challenges, such as multilabel classification, and external validation of these models was insufficient.

Implications of all the available evidence
We believe that AI might advance medical care by improving efficiency of triage to subspecialists and the personalisation of medicine through tailored prediction models. The automated approach to prediction model design improves access to this technology, thus facilitating engagement by the medical community and providing a medium through which clinicians can enhance their understanding of the advantages and potential pitfalls of AI integration. We believe that this Article is novel in its discussion of the scientific and clinical features. Moreover, it addresses health-care delivery aspects and challenges of this technology.
programming interfaces (API) in 2018, claiming to have automated deep learning to such a degree that any individual with basic computer competence could train a high-quality model. 14,15 Because programming is not a common skill among health-care professionals, automated deep learning is a potentially promising platform to support the dissemination of deep learning application development in health care and medical sciences. In the case of classification tasks, these products automatically match generic neural network architectures to a given imaging dataset, fine tune the network aiming at optimising discriminative performance, and create a prediction algorithm as output. In other words, the input is a (labelled) image dataset, and the output is a custom classifying algorithm. Yet, the extent to which people without coding experience can replicate trained deep learning engineers' performance with the help of automated deep learning remains unclear.
In this study, two physicians without any deep learning expertise explored the feasibility of automated deep learning model development and investigated the performance of these models in diagnosing a diverse range of disease from medical imaging. More precisely, we identified medical benchmark imaging datasets for diagnostic image classification tasks and their corresponding publications on deep learning models; used these datasets as input; replaced the classic deep learning models with automated deep learning models; and compared the discriminative performance of the classic and the automated deep learning models. Moreover, we sought to evaluate the interface that was used for automated deep learning model development (Google Cloud AutoML Vision API, beta release) for its use in prediction model research. [16][17][18]

Study design and data source
We used five distinct open-source datasets comprising medical imaging material to automatically develop five corresponding deep learning models for the diagnosis of common diseases or disease features. Namely, we trained deep learning models on retinal fundus images (the MESSIDOR dataset; hereafter referred to as retinal fundus image set); retinal optical coherence tomography (OCT) images (Guangzhou Medical University and Shiley Eye Institute Version 3; hereafter referred to as the retinal OCT set); paediatric chest x-ray (CXR) images (Guangzhou Medical University and Shiley Eye Institute; hereafter referred to as the paediatric CXR set); adult CXR images (National Institute of Health [NIH] CXR14 dataset; hereafter referred to as the adult CXR set); and dermatology images (the Human Against Machine [HAM] 10000 dataset; hereafter referred to as the dermatology image set). [19][20][21][22] Moreover, in a proof-ofprinciple evaluation, we aimed to test one of the models out of sample, as recommended by current guidelines. 23 The current version of the API only allows single image upload for model prediction, limiting the feasibility of large-scale external validation. However, in an effort to emulate external validation in one exemplary use-case, the authors used the dermatology image set for training and tuning of a deep learning model and tested its performance by using a separate skin lesion image dataset, the Edinburgh Dermofit Library (hereafter referred to as the dermatology validation set). 24 Training of health-care professionals using the Graphical User Interface Two physicians (LF and SKW) with no previous coding or machine learning experience did the model development and analysis after a period of self-study. This self-study consisted of basic competence training in shell script programming to allow expedient transfer of the large medical imag ing datasets into a cloudbased Google bucket; familiarisation with the Google Cloud AutoML online documentation and graphical user interface; and preparation of benchmark datasets with randomisation of images to corresponding training, validation, or testing datasets as applicable. In total, each researcher invested approximately 10 h of training time. Because of the release cycle evolution of the Google Cloud AutoML Vision API during the study (alpha release May, 2018, beta release July, 2018), they adopted an iterative approach when executing the analyses. All analytic steps and interpretations of results were done jointly. Interaction with the AutoML Cloud Vision platform was through a graphical user interface (video).

Patient recruitment and enrolment
We accessed five de-identified open-source imaging datasets that were collected from retrospective, nonconsecutive cohorts, showing diseases or disease features of common medical diagnoses. Eligibility criteria, patient demographics and patient workflow for each of these datasets are published elsewhere. [19][20][21][22]

Index test: AutoML Cloud Vision API
The term automated machine learning commonly refers to automated methods for model selection or hyperparameter optimisation. This is the concept that led to the idea of allowing a neural network to design another neural network, through the application of a neural architecture search. [16][17][18] In deep learning, designing and choosing the most suitable model architecture requires a substantial amount of time and experimentation even for those with extensive deep learning expertise. This time and experimentation is because the search space of all possible model architectures can be exponentially large (eg, a typical ten-layer network could have approximately 1 × 10 candidate networks). To make this model design process easier and more accessible, an approach known as neural architecture search has been described. 25 Neural See Online for video architecture search is typically achieved using one of two methods: reinforcement learning algorithms, and evolutionary algorithms. Reinforcement learning algorithms forms the basis of the commercially available API evaluated in this study. 26

Data handling and analytic approach
We uploaded images of five open-source datasets to a Google Cloud bucket in conjunction with a commaseparated value file indicating the image label, file path, and dataset distribution (ie, training, validation, or test dataset) using shell script programming. Upload, and model development and evaluation are shown in the video. Images were allocated to the training, validation, and test datasets (80%, 10%, and 10%, respectively) using a random number function in Microsoft Excel. In the case of the retinal OCT images, where a specific test set had been stipulated in a previous report, we mirrored the same test set to both provide a direct comparison between model performance and to uphold the patient-wise independence between the training and test sets. 20 Duplicate images were automatically detected and excluded by the API. We did not relabel any of the used datasets. All models were trained for a maximum of 24 compute hours. Except for the retinal OCT set, the discriminative performance of each deep learning model was evaluated using the randomly specified test dataset, and in the case of the deep learning model developed on a subset of the dermatology image set, additionally in an external validation using an independent open-source dermatology dataset (Edinburgh Dermofit Library).

Comparison with benchmark classic deep learning models
To be able to make a direct comparison with the performance of classic deep learning models developed using traditional non-autoML techniques (deep learning models with bespoke architectures for a data and problem set developed by human experts), we did a systematic search of the literature to identify classical deep learning models, composed by deep learning experts, which have been trained or validated, or both, on the five open-source datasets. The performance of these existing models served as a direct comparator with the API. We searched MEDLINE, Embase, Science Citation Index, Conference Proceedings Citation Index, Google Scholar, and arXiv from Jan 1, 2012, until Oct 5, 2018. Studies were included if they developed a deep learning algorithm on the datasets used in this study. No language restrictions were applied. The search strategy is available in the appendix (p 1). We prespecified the cut-off of 2012, on the basis of a stepchange in deep learning performance; a deep learning model called AlexNet won a visual recognition challenge, the ImageNet Large-Scale Visual Recognition Challenge, for the first time. 27 If a study provided contingency tables for the same or for separate algorithms tested in a specific classification task, we assumed these to be independent from each other. We accepted this, because we were interested in providing an overview of the results of various studies rather than providing accurate point estimates.

Statistical analysis
The AutoML Cloud Vision API provides metrics that are commonly used by the AI community. These are recall (sensitivity) or precision (positive predictive value) for given thresholds and the area under the precision recall curve (AUPRC). Additionally, confusion matrices are depicted for each model, cross-tabulating ground truth labels versus the labels predicted by the deep learning model. Where possible, we extracted binary diagnostic accuracy data and constructed contingency tables and calculated specificity at the threshold of 0·5. Contingency tables consisted of true-positive, false-positive, truenegative, and false-negative results. If a dataset tackled a multiclass problem, we constructed two-by-two tables for each of the disease labels versus normal. For consistency, we adhered to the typical test accuracy terminology: sensitivity (recall), specificity, and positive predictive value (precision). The classification tasks were chosen according to their popularity in the current AI literature for the purpose of comparability to state-of-the-art deep learning models. Where possible, we plotted contingency tables against the ones reported by other studies using the same benchmark datasets to develop deep learning models.
A priori, we attempted to compare the classification performance between state-of-the-art deep learning studies and our results. However, although the published reports provided areas under the receiver operating characteristic curve (AUC ROC), the AutoML Cloud Vision API reports the AUPRC. Although the points of the two types of curves can be mapped one-to-one and hence curves can be translated from the ROC space to the prediction space (if the confusion matrices are identical) differences in the confusion matrices and the level of reporting impeded us from doing a comparison on the level of AUC. Instead, we compared the performance on the level of sensitivity and specificity at the same threshold as had been used in the previous reports.

Role of the funding source
The funder of the study had no role in study design, data collection, data analysis, data interpretation, or writing of the report. The corresponding author had full access to all the data in the study and had final responsibility for the decision to submit for publication.

Results
We assessed performance of automated deep learning in archetypal binary classification tasks, and found diagnostic properties and discriminative performance were comparable in the case of the investigated binary classification tasks.  There were no duplicate images detected. The AUPRC of the automated deep learning model trained to distinguish these seven categories was 0·93, while best accuracy was reached at a cut-off value of 0·5, with a sensitivity of 91% and a positive predictive value of 91%. We then assessed performance of automated deep learning in multilabel classification tasks, and found the automated deep learning model trained to perform this task on the adult CXR dataset showed poor diagnostic properties and a discriminative performance near chance (AUPRC 0·57, best accuracy at a cut-off value of 0·5, with a sensitivity of 38% and a positive predictive value of 71%). The NIH CXR14 comprised 11 542 cases of atelectasis, 2399 of cardiomegaly, 3323 of consolidation, 1862 of oedema, 8036 of effusion, 1734 of emphysema, 1215 of fibrosis, 156 of hernia, 11 785 of infiltration, 2923 of mass, 3009 of nodule, 1216 of pleural thickening, 325 of pneumonia, and 2199 of pneumothorax, and 60 304 with no findings of 112 120 patients. 12 duplicates were detected and excluded by the API.
We compared the diagnostic properties and the diagnostic performance of algorithms trained using automated deep learning on the retinal fundus image, retinal OCT, paediatric CXR, adult CXR, and dermatology image datasets compared with best performing deep learning algorithms (table 2). Interestingly, all best performing algorithms used transfer learning. Some automated deep learning models showed comparable diagnostic properties at a threshold of 0·5 to state-of-theart deep learning algorithms in published literature. For example, using the OCT dataset, automated deep learning achieved a sensitivity of 97% and a specificity of 100% (vs a sensitivity of 98% and a specificity of 97% published by Kermany and colleagues 20 ); using the paediatric CXR dataset, automated deep learning reached a sensitivity of 97% and a specificity of 100% (vs a sensitivity of 93% and a specificity of 90% published by Kermany and colleagues 20 ). Other models showed lower diagnostic properties; ie, using the multilabel classification task of the NIH CXR14 dataset in which automated deep learning reached sensitivity of 38%, a positive predictive value of 71%, and an AUPRC of 0·57 (vs an AUC of 0·87 published by Guan and colleagues 28 ); or using the retinal fundus image dataset, in which automated deep learning reached a sensitivity of 73% and a specificity of 67% (vs a sensitivity of 86% and a specificity of 97% published by Li and colleagues 29 ). The thresholds were reported in two cases. 28,30 Although it is difficult to determine why multilabel classification model perfor mance was significantly worse, a number of factors might have contributed to this poor performance, including dataset image or labelling ground truth quality, or less likely an inherent weakness of the AutoML platform for multilabel classification.
The AutoML Cloud Vision API provides confusion matrices in the case of single-label classification tasks to uncover label categories in which the model performs insufficiently. The model trained to distinguish the four ophthalmic diagnoses from OCT images (Guangzhou Medical University and Shiley Eye Institute), classified drusen as choroidal neovascularisation (CNV) in 10% of cases, implicating a more urgent referral than needed. The model trained on the dermatology image set on the other hand, misclassified 28·6% of melanomas as naevus, which in a real-world setting would result in less urgent referral for further work-up and delayed, or worse, missed diagnosis. Moreover, this model also had a high misclassification rate (41·7%) for images showing dermatofibromas. Tables 3 and 4 show the corresponding confusion matrices for the OCT and dermatology image set and the figure shows cases from each model where the incorrect label was predicted are shown in tables 3 and 4.
In the case of the deep learning model developed on a subset of the dermatology image set, we additionally did an external validation using the dermatology validation set. As the latter set did not include benign keratosis as a   The automated deep learning model showed poor diagnostic properties and a discriminative performance near coin tossing (AUPRC 0·47, best accuracy at a cut-off value of 0·5, with a sensitivity of 49% and a positive predictive value of 52%). Of note, the sensitivity for melanoma classification is 11% with a misclassification rate of 63·7%.
Interestingly, naevus was the most likely classification in all cases, followed by the ground truth. The only exception was the case of actinic keratosis, where its ground truth diagnosis was the third most probable diagnosis to be predicted (10·6% likelihood). The prevalence of images showing naevus was 76% in the developmental dataset, compared with 36% in the dataset used for external validation. To investigate the effect of class imbalance between the dermatology image set and dermatology validation set, we undersampled the naevus class by 3000 images and assessed the resulting change in accuracy. Only minimal improvements were noted in its discriminative perfor mance and diagnostic properties (data available on request). The external validation of the model trained on a subset of the dermatology image set are shown in tables 5 (discriminative performance and diagnostic properties) and table 6 (confusion matrix).

Discussion
This Article shows that physicians with no coding experience can use automated deep learning to develop algorithms that can do clinical classification tasks to a level comparable with traditional deep learning models that have been applied in existing literature. Most of the automatically developed deep learning models, except for that trained on the multilabel classification task of the adult CXR set, showed comparable discriminative performance and diagnostic properties to state-of-the-art performing deep learning algorithms. The web interface was intuitive to use (video), although a substantial limitation was the inability to batch-test data after the model was created.
From a methodological viewpoint, our results-as is also the case with the results reported in state-of-the-art deep learning studies-might be overly optimistic, because we were not able to test all the models out of sample, as recommended by current guidelines. 23 Moreover, for external validation, the present version of the API only allows single image upload for prediction, limiting large scale external validation. This reduces its usability for systematic evaluation in prediction model research considerably, given the high numbers of images that these datasets comprise.
To circumvent this issue, we emulated an external validation of the model constructed using the dermatology image set and found a substantial reduction in the performance of the deep learning model. The limited performance of automated deep learning models in that case (also in the multilabel classification task) might be related to idiosyncrasies of the datasets used to train the models. To obviate concerns over class imbalance in our external validation, we undersampled accordingly; however, this did not substantially alter the model's discriminative perfor mance. The reasons for the weak performance in the external validation remain unclear: one possibility is the variation in image resolution (although the dermatology validation set images were acquired in a standardised digital fashion, the dermatology   image set was more heterogeneous with a large proportion of digitised images), an issue which can be addressed, to some extent, through image preprocessing. Although the Google Cloud AutoML platform will select an appropriate network architecture, less attention is paid to adjustment of the input data format, such as levels per pixel and image aspect ratio. To our knowledge, this is the first assessment of the feasibility and usefulness of automated deep learning technology in medical imaging classification tasks done by physicians with little programming experience. In this study, we showed best effort to comply with the reporting guidelines for prediction model research, and for developing and reporting machine learning predictive models. 23,31 A strength of the study is that we tested one exemplary model for robustness in an out-of-sample cross validation, because internal validation and randomsplit sample validation has been claimed to overestimate test performance. Another strength is that our results can easily be explored by others, given the use of public datasets and the available free trial use of the AutoML Cloud Vision API.
The sampling used in the in-sample cross validation has been claimed to introduce bias and exaggerate estimates of diagnostic performance. 32 Furthermore, the API was not able to depict saliency maps, and consequently we were not able to interrogate the model for the image areas it considered most important for its prediction. 33 This black box classification does not provide any information useful for clinical purposes. 34 Moreover, the specifics of the models used by the API are not transparent, which constitutes a barrier to their evaluation and the reproducibility of this study. 35,36 We were not able to extract or calculate all metrics and measures of uncertainty conventionally used in prediction model research in all cases (ie, specificity or confidence intervals), which impedes comparison to the current best technology other than deep learning. Moreover, explicit information on how the API determined certain metrics was scarce; for example, it was not clear how the AUPRC for a multiclassification model was calculated. There was also no ability to randomise training, test, and validation groups, while maintaining patient grouping in cases where multiple images might have come from the same patient. We noted that best accuracy was consistently achieved at a threshold of 0·5 by the API. This is likely the default threshold to which the API is optimised: a setting that is inaccessible to the user during model development.
In clinical practice, thresholds should be set according to the role of the diagnostic test and the consequences of a misdiagnosis. Therefore, the ability to adjust the preferred threshold is an important function for creating a fit-forpurpose API. Model interpretability is an active area of research within the field of AI and machine learning. Although possible solutions have been proposed, further work is needed to reach a consensus solution. Such research is outside the scope of this work. 37   Although this report is a proof-of-concept evaluation of health-care professional-led deep learning, it is unclear whether other APIs might provide greater discriminative performance. Assessment of other platforms is an objective of our future research. Currently, studies on AutoML, including ours, have to rely on publicly available datasets. Although using them allows for comparison of performance between different algorithms, these are not without concern. For many classification tasks, and particularly for validation purposes, the existing datasets tend to be too small and not representative of the general population. Moreover, data quality in general could be improved. A full evaluation of dataset limitations is beyond the scope of this Article; however, inconsistent labelling and duplicate images bore direct pertinence to our study. Issues such as equivocal labels (figure), image confounders (presence of chest drain in images of pneumothorax), and label semantics (nodule vs mass, consolidation vs pneumonia) have been noted previously in datasets used for deep learning. 39 Apart from the dermatology image set, all datasets contained duplicate images. Conveniently, Google's AutoML Cloud Vision API will automatically detect, indicate, and exclude the relevant images; however, clinicians need to be cognisant to this possibility, because other APIs might not have this feature and generate spurious evaluation metrics. Because the quality of the results obtained using deep learning models substantially depends on the quality of the dataset used in the model development, it is imperative that patient demographics and information about the way the data was collected (ie, patient flow and whether multiple images from the same patient were present) is presented, because the validity and generalisation of the models would otherwise be difficult to assess. The splitting of test and training when done on a per-patient or per-test set basis might substantially affect model accuracy. However, in our study we were only able to provide limited dataset descriptions, using what has been published by their creators.
With the availability of new and carefully administered datasets, many validity problems could be resolved. Great hopes lay in data-sharing initiatives, as promoted by many peer-reviewed journals, or those from the Dataverse Network Project and the UK Data Archive. 40,41 On the other hand, these initiatives struggle with issues of confidentiality and anonymity when publishing or sharing data relating to individuals. Moreover, regulatory restrictions still remain. Fortunately, developments both in the UK with the NHS Digital Guidance and the call for Health Insurance Portability and Accountability Act compliance 24 in the USA have clarified the framework for many public Cloud systems. The EU General Data Protection Regulation is another possible barrier to an efficient use of published data; however, because many studies will be dealing with ephemeral processing of de-identified data, we do not believe that the General Data Protection Regulation is likely to pose a substantial hindrance.
We confirmed feasibility, but encountered various methodological problems that are well known in research projects performing classification tasks and predictions. We believe that concerted efforts in terms of data quality and accessibility are needed to make automated deep learning modelling successful. Moreover, as the technology evolves, transparency in the reporting of the models and a careful reporting of their performance in methodologically sound validation studies will be pivotal for a successful implementation into practice. Finally, the extent to which automated deep learning algorithms must adhere to regulatory requirements for medical devices is unclear. 35 Although the development of deep learning prediction models was feasible for health-care professionals without any deep learning expertise, we would recommend the following developments for automated deep learning: transparency of the model architectures and technical specifications in use; reporting of established performance parameters in prediction model research, such as sensitivity, specificity reporting of the label distribution in random-split sample validations done automatically (in cases in which the subsets have not been stipulated by the user explicitly); depiction of all incorrectly and correctly classified images (ie, true-positive, false-negative, falsepositive, and true-negative cases); a robust solution to allow systematic external validation; and the addition of granular tools to randomise data splitting for training and validation, while maintaining grouping of images on a patient or visit basis.
The availability of automated deep learning might be a cornerstone for the democratisation of sophisticated algorithmic modelling in health care. It allows the derivation of classification models without requiring a deep understanding of the mathematical, statistical, and programming principles. However, the translation of this technological success to meaningful clinical effect requires concerted efforts and a careful stepwise approach to avoid biasing the results. Deep learning experts and clinicians will need to collaborate in ensuring the safe implementation of artificial intelligence. The sharp contrast of the model's discriminative performance in internal versus external validation might foretell the ultimate use case for automated deep learning software once the technology matures. As researchers and clinicians have excellent access to images and patient data within their own institutions, they might be able to design automated machine learning models for internal research, triage, and customised care delivery. Automating these processes might avert the need for costly external prospective validation across imaging devices, patient populations, and clinician practice patterns. By contrast, large-scale, standardised, deep learning algorithms will necessitate worldwide, multi variable validations of expertly tuned models. Thus, there is considerable value to these small data approaches customised to a specific geographical patient population that a given clinic might encounter. This might be where automated deep learning finds its niche in the medical field. Importantly, this could make models susceptible to selection bias, overfitting, and a number of other issues from imprecise model training or patient selection. Novices should adhere to ethical principles when designing these models to avoid discrimination and causing harm. 42,43 Therefore, regulatory guidelines are needed for both medical deep learning and clinical implementation of these models before they might be used in clinical practice. In summary, although our approach seems rational in this early evaluation, the results of this study cannot yet be extrapolated into clinical practice.


Boosting LiDAR-based Semantic Labeling by Cross-Modal Training Data Generation

Mobile robots and autonomous vehicles rely on multi-modal sensor setups to perceive and understand their surroundings. Aside from cameras, LiDAR sensors represent a central component of state-of-the-art perception systems. In addition to accurate spatial perception, a comprehensive semantic understanding of the environment is essential for efficient and safe operation. In this paper we present a novel deep neural network architecture called LiLaNet for point-wise, multi-class semantic labeling of semi-dense LiDAR data. The network utilizes virtual image projections of the 3D point clouds for efficient inference. Further, we propose an automated process for large-scale cross-modal training data generation called Autolabeling, in order to boost semantic labeling performance while keeping the manual annotation effort low. The effectiveness of the proposed network architecture as well as the automated data generation process is demonstrated on a manually annotated ground truth dataset. LiLaNet is shown to significantly outperform current state-of-the-art CNN architectures for LiDAR data. Applying our automatically generated large-scale training data yields a boost of up to 14 percentage points compared to networks trained on manually annotated data only.

Introduction
Within the fields of mobile robotics and autonomous driving, vehicles are typically equipped with multiple sensors of complementary modalities such as cameras, LiDAR and RADAR in order to generate a comprehensive and robust representation of the environment [2][3][4][5]. Each sensor modality leverages its specific strengths to extract as much information as possible from the observed scene. Based on the combined sensor data, a detailed environment model, for example in the form of a dynamic occupancy grid map [6], is created. This environment Fig. 1. Example of a LiDAR point cloud labeled by LiLaNet. Point colors correspond to the Cityscapes semantic class color coding [1]. The test vehicle is headed to the top right, the corresponding camera image is shown on the top left for clarity. model provides the basis for high-level tasks such as object tracking [7], situation analysis and path planning [8]. In order to master these high-level tasks, it is essential for an autonomous vehicle to not only distinguish between generic obstacles and free-space, but to also obtain a deeper semantic understanding of its surroundings. Within the field of computer vision, the corresponding task of semantic image labeling has experienced a significant boost in recent years due to the resurgence of advanced deep learning techniques [9]. However, detailed semantic information of similar quality has to be extracted independently from each of the sensor modalities to maximize system performance, availability and safety. Therefore, in this paper we introduce LiLaNet, an efficient deep neural network architecture for point-wise, multi-class semantic labeling of semi-dense LiDAR data.
As has been well established within the camera domain, large-scale datasets are of paramount importance for training competitive deep neural networks. Consequently, large-scale generic datasets such as ImageNet [10] and COCO [11], as well as medium-scale datasets dedicated to road scenarios such as KITTI [12] and Cityscapes [1] have been made available for this purpose. In contrast, within the LiDAR domain only indoor datasets [13][14][15] or outdoor datasets [16] obtained from high-resolution stationary sensors have been published to date. For this reason some authors [17][18][19] have resorted to the indirect extraction of the desired LiDAR point semantics from the KITTI dataset, using annotated object bounding boxes or the camera road detection benchmark. While this type of indirect extraction eventually yields viable training datasets, it is relatively cumbersome and limited to only a small set of semantic classes. Hence, here we propose a so-called Autolabeling process, an effective approach for the automated generation of large amounts of semantically annotated mobile LiDAR data by the direct transfer of high-quality semantic information from a registered reference camera image. The semantic information in the reference image is obtained using an off-the-shelf neural network. We show that the datasets obtained with this approach significantly boost the LiDAR-based semantic labeling performance, in particular when augmented with a small manually annotated dataset for finetuning.
Our main contributions can be summarized as follows: 1. An efficient Convolutional Neural Network (CNN) architecture for highquality semantic labeling of semi-dense point clouds, as provided by stateof-the-art mobile LiDAR sensors. 2. A large-scale automated cross-modal training data generation process for boosting the LiDAR-based semantic labeling performance. 3. A thorough quantitative evaluation of the semantic labeling performance, including an analysis of the proposed automated training data generation process.

Related Work
LiDAR-based semantic labeling has gained increased attention in recent years due to the availability of improved mobile sensor technology, providing higher resolution and longer range at reduced cost. The various proposed approaches of LiDAR-based semantic labeling can be discriminated by the way the point-wise 3D information is utilized.
The first approaches using depth information for semantic labeling were based on RGB-D data, which complements RGB image data with an additional depth channel [20,21], allowing to recycle 2D semantic image labeling algorithms. Often a stereo camera was used to create a dense depth image, which was then fused with the RGB image. Tosteberg [22] developed a technique to use depth information of a LiDAR sensor accumulated over time to project it into the camera space. The accumulation yields a depth image of increased density without requiring dedicated upsampling algorithms.
A different category of approaches considers the 3D LiDAR data as an unordered point cloud, including PointNet [23], PointNet++ [24] and PointCNN [25]. The PointNet architecture [23] combines local point features with globally extracted feature vectors, allowing for the inference of semantic information on a point-wise basis. Extending this idea, PointNet++ [24] introduces a hierarchical PointNet architecture to generate an additional mid-level feature representation for an improved handling of point neighborhood relations. Both approaches are evaluated successfully on indoor scenes but reach their limits in large scale outdoor scenarios. The PointCNN [25] approach is based on unordered point clouds as well, but introduces modified convolution layers extended by permutations and weighting of the input features. This allows to transfer the advantages of traditional CNNs to unordered point cloud processing. However, the approach is only used for object detection and has not yet been applied to semantic labeling of point clouds.
Yet another way of representing LiDAR input data is within cartesian 3D space, which is used in the SEGCloud [26] and OctNet [27] methods. Here a Voxel (SEGCloud) or an OctTree (OctNet) representation is created and the convolution layers are extended to 3D convolutions. These approaches retain the original 3D structure of the input points, making them more powerful in preserving spatial relations. However, the algorithms have to cope with the high sparsity of the data, and inference time as well as memory requirements increase drastically for large-scale outdoor scenes.
A possible solution to avoid the computational complexity of 3D convolutions is the rendering of 2D views of the input data. Based on such 2D views, state-ofthe-art image-based deep learning algorithms can be applied. Depending on the use case, different viewpoints or virtual cameras may be used. Caltagirone et al. [19] use a top-view image of a LiDAR point cloud for labeling road points within a street environment. This top-view projection of the LiDAR points is a valid choice for road detection, but the resulting mutual point occlusions generate difficulties for more general semantic labeling task as in our case. An alternative is to place the virtual camera origin directly within the sensor itself. The resulting 2D view is often visualized via a cylindrical projection of the LiDAR points (see Fig. 2), which is particularly suitable for the regular measurement layout of common rotating LiDAR scanners. In this case, the sensor view provides a dense depth image, which is highly advantageous for subsequent processing steps. Wu et al. [17] uses this type of input image for the SqeezeSeg architecture, which performs a SqeezeNet-based [28] semantic labeling to segment cars, pedestrians and cyclist. The bounding boxes of the KITTI object detection dataset [12] are used to transfer the point-wise semantic information required for training and evaluation. The approach of Dewan et al. [18] uses the cylindrical projection of the LiDAR data as an input for a CNN based on the Fast-Net architecture [29] to distinguish between movable and non-movable points. Similar to Wu et al [17], the KITTI object detection dataset is used for transferring the ground-truth bounding box labels to the enclosed points.
Varga et al [30] propose an alternative method to generate a semantically labeled point cloud at runtime, based on a combined setup of fisheye cameras and LiDAR sensors. First, pixel-wise semantics are extracted from the camera images via a CNN model trained on Cityscapes [1]. Subsequently, the LiDAR points are projected into the images to transfer the semantic information from pixels to 3D points. However, no semantic information is inferred on the LiDAR point cloud itself, and spatial and temporal registration of the sensor modalities remains a challenge. In the present paper we take the idea of [30] one step further and utilize a joint camera/LiDAR sensor setup to generate large amounts of 3D semantic training data. The data is then used to train a deep neural network to infer pointwise semantic information directly on LiDAR data. We show that combining the large amounts of automatically generated data with a small manually annotated dataset boosts the overall semantic labeling performance significantly.

Method
We introduce a novel CNN architecture called LiLaNet for the point-wise, multiclass semantic labeling of LiDAR data. To obtain high output quality and retain efficiency at the same time, we aim to transfer lessons learned from image-based semantic labeling methods to the LiDAR domain. The cylindrical projection of a 360 • point cloud captured with a state-of-the-art rotating LiDAR scanner is used as input to our networks. Training is boosted by an efficient automated cross-modal data generation process, which we refer to as Autolabeling.

LiDAR Images
LiDAR sensors measure the time of flight of emitted laser pulses in order to determine the distance of surrounding objects with high accuracy. In addition, modern sensors even provide coarse reflectivity estimates on a point-wise basis. For the following experiments we consider a Velodyne VLP32C LiDAR scanner, which features 32 vertically stacked send/receive modules rotating around a common vertical axis. While rotating, each module periodically measures the distance and reflectivity at its current orientation, i.e. at the respective azimuth and elevation angles. We combine the measurements of a full 360 • scan to create cylindrical depth and reflectivity images, as illustrated in Fig. 2. This projection represents the view of a virtual 360 • cylindrical camera placed at the sensor origin. At ten revolutions per second, images of size 1800×32 pixels are obtained.
The cylindrical point cloud projection provides dense depth and reflectivity images which are free from mutual point occlusions. This allows for the application of optimized 2D convolution layers, as used with great success in state-of-the-art image-based CNN architectures. In this way, inference time is reduced drastically compared to the use of full 3D input representations such as voxel grids or octtrees. Further, since measurement times and orientation angles are known with high accuracy, it is straightforward to transform the cylindrical image back into a full three-dimensional point cloud representation without any loss of information.
In cases where no laser reflection is received by the sensor, for example when pointed towards the sky, pixels corresponding to the respective measurement angles are marked as invalid in the resulting depth image.

Class Mapping
Based on initial experiments we conducted with regard to the discrimination of different semantic classes in LiDAR data, we first apply a mapping of the original Cityscapes labelset [1] to a reduced set of 13 semantic classes (see Table 1). This reduced label set is better tailored to the specific properties of the data provided by current LiDAR sensors, where limited spatial resolution and coarse reflectivity estimates prohibit truly fine-grained semantic differentiation. For example, the original 'truck' and 'bus' classes are merged into a common 'large vehicle' class. Similarly, the original 'motorcycle' and 'bicycle' classes are combined into a single Fig. 2. Example of a depth image (center, blue = close, red = far) and reflectivity image (right, black = non-reflective, cyan = highly reflective) resulting from the cylindrical point cloud projection. The corresponding camera image is shown on the left. Note that here the 360 • LiDAR scan has been cropped to the camera field of view for illustration. 'two wheeler' class. The 'fence' class is not retained in our mapping, as such thin and porous structures are hardly captured by LiDAR sensors. Note that the reduced label set still provides an abundance of valuable semantic information with adequate detail, which is highly beneficial for a large set of application scenarios.

LiLaNet Network Architecture
Using the LiDAR images described in Section 3.1 as input, we present a dedicated CNN architecture for high-quality LiDAR-based semantic labeling. To cope with the low resolution and extreme asymmetry in the aspect ratio of the used LiDAR images, we propose a dedicated network block called LiLaBlock, which is inspired by the GoogLeNet inception modules of [31]. The block structure is illustrated in Fig. 3. In order to successfully handle relevant objects of various aspect ratios, the LiLaBlock applies convolution kernels of sizes 7×3, 3×7 and 3×3 in parallel. The output is then concatenated and the dimension is decreased by a factor of three via a bottleneck. In this way the dimensionality of the feature space is reduced yielding a more compact representation. At the same time the inference complexity of the LiLaNet is lowered. Note that each convolution is followed by a rectified linear unit layer (ReLU layer).  Fig. 3. The LiLaBlock structure allows to cope with the extreme asymmetry in the aspect ratio of the input LiDAR images.

Reflectivity
Concat LiLaBlock The full LiLaNet consists of a sequence of five LiLaBlocks with a varying number of kernels, as shown in Fig. 4. The two input channels represent the concatenated depth and reflectivity images, while the output provides the corresponding point-wise semantic image labeling, according to the label set defined in Section 3.2.
The network training is performed via the Adam solver [32]. We use the suggested default values of β 1 = 0.9, β 2 = 0.999 and = 10 −8 . The learning rate is fixed at α = 10 −3 (α = 10 −4 for fine-tuning) and the batch size is set to b = 5, while the weights are initialized with MSRA [33].

Autolabeling
Generating manually annotated training data for LiDAR-based semantic labeling at scale presents a huge effort and entails even higher cost compared to manual image annotations in the 2D domain. This is due to both the additional spatial dimension and the sparsity of the data, which results in a representation that is non-intuitive and cumbersome for human annotators. For these reasons, we introduce an efficient automated process for large-scale training data generation called Autolabeling.
As illustrated in Fig. 5, the Autolabeling concept is based on the use of one or more reference cameras in conjunction with the LiDAR sensor capturing the point cloud data. The obtained reference camera images have to be registered to the LiDAR data in space and time. Preferably, the spatial displacement between the sensor origins is minimized to avoid occlusion artifacts. In the first step, a high-quality pixel-wise semantic labeling of the reference camera image is computed via state-of-the-art deep neural networks, as can be found on the leaderboard of the Cityscapes benchmark [1]. Second, the captured point cloud is projected into the reference image plane to transfer the semantic information of the image pixels to the corresponding LiDAR points. While a single reference camera will in general only cover a fraction of the full point cloud, it is straightforward to extend the approach to multiple cameras for increased coverage.
The described fully automated procedure yields semantically labeled point clouds which can directly be used to train LiDAR-based semantic labeling networks such as LiLaNet. In the following subsections we describe the various stages of the data generation process in more detail.

Semantic Image Labeling
For the experiments in the present paper we use an efficient reference network as described in [34] to obtain the pixel-wise semantic labeling of the camera images. The network is trained on the Cityscapes dataset and achieves an Intersection-over-Union (IoU) test score of 72.6% with regard to the original Cityscapes label set. Since the Cityscapes dataset was designed with vehicle-mounted front-facing cameras in mind, we also use a single front-facing camera to evaluate the proposed automated training data generation process.
Note that the Autolabeling process can be applied using any image-based reference network providing sufficient output quality. Moreover, the process will in general directly benefit from the ongoing research and improvements in imagebased semantic labeling networks.

Point Projection
In order to project the 3D points captured by a scanning LiDAR into the reference camera image plane, several aspects have to be taken into account. Since the LiDAR scanner rotates around its own axis in order to obtain a 360 • point cloud, each measurement is taken at a different point in time. In contrast, the camera image is taken at a single point in time, or at least with a comparatively fast shutter speed.
To minimize potential adverse effects introduced by the scanning motion of the LiDAR, we apply a point-wise ego-motion correction using vehicle odometry.  6. Example of the point projection with (right) and without (left) ego-motion correction. The colors of the semantically labeled images represent the Cityscapes label set, while the projected points are color-coded according to distance (dark green = far away, yellow = close). Note that invalid LiDAR points are not projected into these images.
First, the measured 3D points are transformed from the LiDAR coordinate system to the vehicle coordinate system via the extrinsic calibration parameters of the sensor. Given the points p v = (x v , y v , z v ) in the vehicle coordinate system, the wheel odometry data is used to compensate for the ego-motion of the vehicle. To this end, the time difference ∆t between the point measurement timestamp t p and the image acquisition timestamp t c of the camera is computed for each point. In case of a rolling shutter camera, half of the shutter interval t r is added to move the reference timestamp to the image center: The time difference ∆t is used to extract the corresponding ego-motion data of the vehicle from the odometry sensor. This yields a transformation matrix T ∆t describing the motion that occurred between the two timestamps of interest. Using the transformation matrix T ∆t , each point This effectively transforms each point p v in the vehicle coordinate system to its corresponding position p t c = (x t c , y t c , z t c ) at camera timestamp t c . Finally, the corrected points p t c are transformed to the camera coordinate system and projected to the image plane using a pinhole camera model [35]. An exemplary result of the point projection with and without ego-motion correction is shown in Fig. 6. Fig. 7. Illustration of the timing problem (top view), where the camera (orange) is mounted on top of the LiDAR (blue). At a given timestamp, the rotating LiDAR measures points only at the angle α h (blue line). The tree (green) is detected by the LiDAR at timestamp t c −∆t, while the camera captures an image of the tree at t c . This time difference ∆t results in a spatial displacement of the sensor origins, which causes occlusion artifacts resembling a large mounting distance between the two sensors.

Dataset
To train LiLaNet, a large dataset was created based on the Autolabeling process described Section 3. Approximately 555,000 frames of different road types (cities, rural roads, highways) were recorded and automatically labeled. Subsequently the recorded frames were split into subsets for training, validation and testing, with a split of approximately 62% -13% -25%. Details are listed in Table 2. The dataset split was performed on a sequence basis instead of via random selection in order to prevent correlations in between subsets. The training dataset was used to train LiLaNet, while the validation dataset was used to optimize the network architecture (not shown within this publication). The testing dataset is used to evalute the achieved LiDAR-based semantic labeling performance and the benefits of the proposed Autolabeling process. An essential factor for accurate Autolabeling results is a small spatial distance between the LiDAR sensor and the reference camera, as occlusion artifacts tend to introduce inconsistencies to the datasets. This can be accommodated by both a small mounting distance of the two sensors and explicitly taking time synchronization into account. In the used setup, the camera is not being triggered by the LiDAR sensor, which results in cases where the LiDAR sensor orientation is not well aligned with the front-facing camera during image acquisition time. This misalignment causes a significant timestamp difference between the projected point cloud and the image data. While this effect is compensated by the ego-motion correction described in Section 3.4, it usually results in a significant translatory motion between the two sensors while driving, which in turn leads to considerable occlusion artifacts. See Fig. 7 for an illustration of the problem. For this reason, we define a maximum heading deviation range γ h = 60 • which is allowed between the camera principal axis orientation α c and the current LiDAR azimuth angle α h at the camera time stamp t c . All captured frames which lie outside of this maximum deviation range are discarded, yielding an optimized subset of the originally recorded data (see Table 2).
Based on the optimized datasets, 3,000 keyframes were selected equally distributed across all frames for manual annotation. This manually annotated data forms the basis for the evaluation in Section 5. Note that invalid LiDAR points are not being annotated and hence the class 'sky' is not considered for the evaluation in the present paper.

Experiments
The Autolabeling process introduced in Section 3.4 was applied to automatically generate the large-scale dataset described in Section 4, which in turn was used to train the proposed LiLaNet architecture for LiDAR based semantic labeling. In this section we evaluate different training strategies for LiLaNet in detail. In particular, we analyze the impact of boosting the training via the large-scale datasets obtained from the Autolabeling process. Note that all evaluations are based on the testing subset of the manually annotated keyframes. Following the Cityscapes Benchmark Suite [1], we apply the Intersection-over-Union (IoU) metric for performance evaluation. We define the following evaluation schemes, which are also visualized in Fig. 8: (1) Autolabeling: This evaluation considers inaccuracies of the image CNN as well as projection errors. No LiDAR-based CNN is trained for this evaluation. (2) LiLaNet Manual Annotations: This evaluation assesses the performance of the LiDAR-based semantic labeling using a small set of cost-intensive manually annotated point clouds from the annotated keyframes for training (see Table 2).

(3) LiLaNet Autolabeled Reduced:
This evaluation measures the performance of the LiDAR-based semantic labeling using a small set of automatically generated training data, based on the annotated keyframes (same keyframes that constitute the training subset of the manually annotated dataset shown in Table 2). (4) LiLaNet Autolabeled Full: This evaluation assesses the performance of the LiDAR-based semantic labeling when using the Autolabeling process on the full training dataset of the original frames (see Table 2). (5) LiLaNet Finetuned: This evaluation measures the performance of the LiDAR-based semantic labeling by fine-tuning the network using a small set of manually annotated data (2) with a pre-training based on the full Autolabeling process (4).  Fig. 8. Overview of the various training strategies, which differ in the amount of training data (see Table 2) as well as the labeling type of the ground truth.  Table 3. Class-wise and overall IoU scores of the different approaches. The highest IoU scores of each column (excluding the first row) are marked in bold.
The used image-based semantic labeling network (Section 3.4) was previously analyzed in detail in [34], achieving an IoU score of 72.6% on the Cityscapes Benchmark Suite. In addition, we re-evaluate the image-based approach on the Cityscapes test set when using the class mapping defined in Section 3.2. This yields an IoU score of 79.3%, detailed results are listed in Table 3. Compared to the pure image-based semantic labeling result, the Autolabeling (1) output used to generate our large-scale dataset shows worse performance scores. This may be attributed to various reasons. First of all, the semantic image labeling CNN is optimized towards the specific properties of high-resolution camera data, which may lead to a slight reduction in performance when transferring the labels into the LiDAR domain. Also, despite the ego-motion correction efforts described in Section 4, the remaining inaccuracies still result in a certain amount of occlusion artifacts. Finally, a multi-sensor setup is rarely free from small calibration offsets in practice, even if optimized manually. These imperfections cause projection  misalignments of the points within the images, which results in inaccurate label assignments, in particular for points at a large distance to the sensors.
The detailed results of the various training strategies applied to LiLaNet are listed in Table 3 and illustrated in Fig. 9. It can be seen that the training on manually annotated data (2) yields a higher performance than the training on autolabeled data (3), but only as long as the same amount of data is being used. This is due to the imperfect results of the Autolabeling output itself. However, as stated previously, in practice the amount of available manually annotated data is severely limited by the high cost of point-wise manual annotation. In contrast, the Autolabeling process allows to automatically generate training data sets of arbitrary size at low cost. When using the large amount of automatically generated data for training (4), LiLaNet in fact outperforms its respective variant trained on manual annotations (2) by 4.6 percentage points with regard to mean IoU. Moreover, the network seems to generalize well within the LiDAR domain and suppresses some errors introduced by the Autolabeling process, which is indicated by the improved performance for some classes when compared to the pure Autolabeling output (e.g. 'construction', 'pole', 'traffic sign'). Furthermore, Fig. 9 shows that the training on the small amount of data in (2) and (3) saturates after several thousand iterations, while the training on the large-scale dataset (4) continues to increase output performance. Finally, using the manual annotations to fine-tune LiLaNet after pre-training on the automatically generated dataset (5) boosts performance by another 9.5 percentage points. This corresponds to a total gain of 14.1 percentage points over the training on manually annotated data only. Note that after fine-tuning most classes achieve a significantly higher performance than obtained by the pure Autolabeling output itself. Hence, the LiDAR-based semantic labeling result provided by LiLaNet with a training supported by the Autolabeling process outperforms the imagebased semantic labeling results projected into the LiDAR domain. It is worth noting that the network fine-tuning reaches its maximum performance only after a few thousand iterations and soon starts to overfit on the small manually annotated dataset, as can be seen in Fig. 9. A qualitative result of the fine-tuned network is shown in Fig. 1.
In order to compare the presented LiLaNet architecture to the state-of-theart, we also analyze the performance of the SqueezeSeg architecture [17], recently proposed for LiDAR-based semantic labeling on a smaller set of semantic classes ('car', 'pedestrian', 'cyclist'). Note that we evaluate the SqueezeSeg approach without its Conditional Random Field (CRF) stage for a fair comparison. The results in Table 3 as well as Fig. 9 illustrate that LiLaNet outperforms the SqueezeNet architecture in each class. This may be due to the following reasons: First of all, SqueezeNet uses five horizontal pooling layers which increases the learnable feature size drastically. Consequently, the network may have difficulties in capturing small but relevant objects. Further, the SqueezeSeg architecture does not distinguish between different object shapes and sizes in the design of the convolution kernels, as is done in our LiLaBlock structure. However, note that SqueezeNet does indeed also benefit from our combined process of fine-tuning after pre-training on an automatically generated dataset.

Conclusion
Autonomous vehicles require a comprehensive and robust environment representation, including a detailed semantic understanding of their surroundings. This can be obtained by a combination of different sensor modalities, where each sensor independently contributes to the overall environmental model. In the present paper we consider the point-wise multi-class semantic labeling of 3D point clouds and transfer the concept of pixel-wise image-based semantic labeling to the LiDAR domain. We propose LiLaNet, a novel CNN architecture for efficient LiDAR-based semantic labeling. This architecture significantly outperforms current state-of-the-art CNNs for multi-class LiDAR-based semantic labeling when evaluated on a manually annotated ground truth dataset.
Furthermore, we present a fully automated process for large-scale cross-modal training data generation called Autolabeling. The approach is based on the use of reference cameras in order to transfer high-quality image-based semantic labeling results to LiDAR point clouds. Combining the automatically generated training dataset with a fine-tuning step based on small-scale manually annotated data yields a performance boost of up to 14 percentage points while keeping manual annotation efforts low.