Parallel Closed-Loop Connected Vehicle Simulator for Large-Scale Transportation Network Management: Challenges, Issues, and Solution Approaches

The augmented scale and complexity of urban transportation networks have significantly increased the execution time and resource requirements of vehicular network simulations, exceeding the capabilities of sequential simulators. The need for a parallel and distributed simulation environment is inevitable from a smart city perspective, especially when the entire city-wide information system is expected to be integrated with numerous services and ITS applications. In this paper, we present a conceptual model of an Integrated Distributed Connected Vehicle Simulator (IDCVS) that can emulate real-time traffic in a large metro area by incorporating hardware-in-the-loop simulation together with the closed-loop coupling of SUMO and OMNET++. We also discuss the challenges, issues, and solution approaches for implementing such a parallel closed-loop transportation network simulator by addressing transportation network partitioning problems, synchronization, and scalability issues. One unique feature of the envisioned integrated simulation tool is that it utilizes the vehicle traces collected through multiple roadway sensors-DSRC onboard unit, magnetometer, loop detector, and video detector. Another major feature of the proposed model is the incorporation of hybrid parallelism in both transportation and communication simulation platforms. We identify the challenges and issues involved in IDCVS to incorporate this multi-level parallelism. We also discuss the approaches for integrating hardware-in-the-loop simulation, addressing the steps involved in preprocessing sensor data, filtering, and extrapolating missing data, managing large real-time traffic data, and handling different data formats.

I. INTRODUCTION
ith the advent of big data and connected vehicle (CV) technologies, the parameters and requirements for simulating metro-scale urban transportation networks with heterogeneous vehicles have evolved substantially. Today's transportation engineers at the Traffic Management Centers (TMCs) feel the necessity of a parallel CV simulation tool that could allow them to visualize the immediate systemwide effect of any change in traffic parameters-signal timing, detour, lane closure-before making the decisions. Unfortunately, current state-of-the-art traffic simulators (VISSIM [1], CORSIM [2], SUMO [3] etc.) are designed and developed primarily for the microscopic simulation of vehicles and pedestrians, evaluation of traffic control algorithms, and visualization of on-road traffic behaviors. Since these traffic simulators were not originally designed to model the multilayer wireless network protocols (IEEE 802.11p, IEEE 1609.x) required to simulate the vehicle-to-vehicle (V2V) and vehicleto-infrastructure (V2I) communications, additional work is necessary to port and feed the network simulators with these traffic simulators to simulate a transportation network with connected vehicles. Simulation of a transportation network with CV requires a bi-directional coupling mechanism between a transportation simulator and a communication simulator. This mechanism has led to the concept of the closed-loop CV simulator, which has recently drawn a significant amount of research interests within the community. However, the computational capacity of such a bi-directionally coupled (closed-loop) simulator is significantly limited by the number of CVs equipped with onboard units (OBUs) and the number of roadside units (RSUs) deployed within the metro-wide transportation network, since these DSRC devices transmit millions of basic safety messages (BSMs) packets every minute requiring massive computational resources. Existing sequential closed-loop simulators can barely handle one thousand vehicles simulated in a scenario involving no more than a few intersections. Thus, the incorporation of parallelism in both transportation and communication simulation platforms will enable efficient management of large-scale transportation network and control of traffic parameters involving connected vehicles. In addition, the integration of roadway sensor data through hardware-in-the-loop simulation with the closed-loop software simulator will enable the traffic engineers to make informed decisions by evaluating the system-wide impact of traffic parameters changes in real-time. The hardware-in-theloop simulation (HILS) module in a parallel computing environment will facilitate system-wide visualization of traffic status from the Transportation Management Center (TMC). A potential use case for our HILS-incorporated parallel CV simulation tool is to provide a realistic prediction of the consequences of traffic change-such as transit bus delays or tentative queue length considering the preemptive detour advisory disseminated through DSRC-enabling a traffic engineer to make the real-time decision when a major corridor needs to undergo closure of lanes due to maintenance. Another important benefit of integrating HILS is to validate the simulation results of V2I applications.
A vast amount of research effort has been recently directed towards the improvement of surface transportation through selfdriving autonomous vehicles as well as connected vehicles (CVs) using the 5.9 GHz Dedicated Short Range To accurately simulate ITS applications on a scenario involving connected vehicles, it is necessary to integrate a full-fledged transportation simulator with a wireless network simulator, resulting in the need for a closed-loop simulator. This kind of closed-loop simulator requires a tight synchronization between two stand-alone simulation modules: a transportation module and a communication module. The transportation module is responsible for the modeling of vehicle mobility applications including traffic routing, car-following, lane-changing, vehicle dynamics, driver behavior modeling, and traffic signal control modeling etc. On the other hand, the communication module accounts for data traffic network modeling including packet routing, end-to-end V2X packet delivery, wireless media access, cross-layer protocols, information security, and authentication mechanisms. In a CV simulation environment, the two simulation modules (transportation and communication) operate as a real-time feedback control loop with a tight synchronization. These two modules highly influence the operations of one another. For example, vehicle dynamics, mobility, speed, and density affect the communication links between vehicles as well as the data packet routing; hence, they also affect the communication quality, i.e., reliability, throughput, and delay. Conversely, the data communication parameters-for example, the number of packet losses between vehicles and the end-to-end delivery delay-can adversely affect the mobility decisions made by the transportation simulator, particularly when a V2X message carries detour information due to an accident. For a V2X-based safety application, it is important to realize that the slightest delay in communication, even about a fraction of a second, can have serious consequences and may even be fatal. Considering the complexity of the transportation and communication module with the high level of interdependency between them, it is easy to perceive how challenging the simulation of an integrated CV system can be.

II. RELATED WORK
Researchers have been focusing to develop a complete feedback-loop based transportation simulator for the past decade. More specifically, they are putting their effort to develop a transportation network with a wireless network simulator for simulating V2X-based ITS applications. Many researchers studied sequential simulators, but a comparative modular analysis of different simulator components is still needed for identifying the capabilities and limitations of the simulators. Thus, we focus on several sequential simulators to compare their modular organization and architecture.
Early efforts to simulate vehicular networks were based on fixed mobility trajectories that were fed to the network simulators. Contemporary network simulators are designed to simulate communication protocol and hence, do not have the capability to generate a realistic car-following model. Thus, there was a need for ingestion of fixed mobility traces for the communicating nodes. Several mobility generator frameworks (VANETMOBISIM [7], SUMO [3], MOVE [8], STRAW [9], FREESIM [10], CITYMOB [11], C4R [38]) have been developed to produce the vehicular trajectories that are fed into various network simulators (NS2 [12], NS3 [13], OMNET++ [14], OPNET [15], JIST/SWANS [16], QualNET [17], etc.). However, ingesting static mobility traces into these network simulators could not incorporate the effect of ITS applications on the mobility of the vehicles. a virtual traffic network simulation mode, a playback mode, a hybrid simulation mode, and a test scenario generation mode. Automesh [64] supports external modules integration with plug and play capabilities. However, the framework was merely proposed but not implemented. STRAW [9] utilizes their own traffic model for simulating intra-segment and intersegment vehicle motion.

Application Focused Simulation
Some researchers have been focusing on developing simulation frameworks for simulating intelligent traffic controls for autonomous vehicles. For example, Gelbal et al. present a hardware-in-the-loop (HiL) simulator capable of running lane keeping and adaptive cruise control car following applications [65]. The HiL simulator consists of a dSPACE Scalexio system, a dSPACE Microautobox control unit, DSRC radios, and sensors. The dSPACE Scalexio system runs the Carsim simulator with real-time traffic and sensor data. The dSPACE Microautobox control unit provides the autonomous functionalities. So, the HiL simulator focuses more on the application side of a simulator rather than the internal modular design.
Some researchers also focus on the application side of a connected vehicle simulator. Luigi et al. compare the spacing adopted in equilibrium car-following conditions involving a real driver in the loop simulator using driving simulators and instrumented cars [66].
Many researchers have been trying to integrate simulators in different domains to package those simulators into a single simulation framework. For example, Zhao et al. propose a 3-1 integrated traffic-driving-networking simulator (ITDNS) [67]. The purpose of ITDNS is to simulate cyber transportation system and connected vehicle applications allowing a human driving into the virtual simulation environment. The authors in [68] discuss a framework for simulating a road traffic control policy to reduce the waiting time for emergency vehicles. To do so, they interconnect a multi-agent system development framework (JADE) and an agent-based traffic simulator (SUMO) using TraSMAPI [69]. IsV2C, an integrated road traffic-network-cloud simulator [70], aims to simulate the vehicle-to-cloud services for providing real-time performance for applications such as driving assistance, infotainment, and vehicle maintenance. Another service-oriented simulator, QoS-CITS [71] provides researchers ways to conduct various experiments including parameter tuning for their study. The authors in [72] discuss Similitude, a framework for simulating traffic scenarios, network communications, and ITS mobile applications. They use SimMobility as the traffic simulator, ns-3 as the network simulator, and QEMU as the mobile application (Android) simulator.
Many researchers also enhance existing traffic and network simulators to cover a wide range of traffic scenarios and communication mediums. For example, researchers in [73] focus on simulating network communications in heterogeneous networks keeping in mind to support new communication mediums as well as the legacy mediums. More specifically, they extend ns-3 to support the communication simulation in visible light communications (VLC) medium. Lim et al. enhance SUMO for meeting requirements for the roadway scenario generation for different transportation system (e.g., left-hand side simulation or right-hand side simulation) [74]. Abeywardana et al. extend veins to add support for advertising cognitive radio-based services [75].

Open-Loop Simulators
Lee and Park [18] used the NCTUns communications simulator to examine the effects of communications using VISSIM trajectory data offline with no feedback loop for traffic simulation. GrooveSim [19] simulates inter-vehicular communication and vehicular mobility in a road traffic network using a customized mobility model and the GrooveNet [20] routing protocol. MobiREAL [21] incorporates mobility support on the Georgia Tech Network Simulator (GTNetS [22]). The capabilities of these type of open-loop simulators are limited to studying only unidirectional effects between the two domains. For example, studying the effect of various mobility models on the performance of end-to-end data communication using these simulators could characterize the dependency of the communication module on the transportation module, but it would be impossible to study the impact of data communication on the transportation system by incorporating changes in vehicle route, speed, signal timings, and mobility patterns based on newly received messages. Hence, this approach cannot be used to study bidirectional effects between the two tightly coupled domains.

Closed Loop Simulators
Recently, there has been a significant amount of interests and efforts to design closed-loop CV simulators by coupling transportation and wireless network simulators. These closedloop simulators can be further divided into two types-1) Fixed pair coupling and 2) Flexible pair coupling.

1) Fixed Pair Coupling
Most closed loop CV simulators are fixed pair, which means that the choice of transportation simulator and communication simulator is already decided based on the CV simulator. For example, Traffic and Network Simulation Environment (TraNS [23]) links SUMO and NS-2. Multiple Simulator Interlinking Environment for IVC (MSIE [24]) integrates NS-2, VISSIM traffic simulation, and application simulation (MATLAB) into a simulation environment for vehicular ad hoc networks (VANETs). Veins [4] provides a closed-loop integration using SUMO and OMNeT++ as the traffic and communication simulators respectively. Integrated Wireless and Traffic Platform for Real-Time Road Traffic Management Solutions (iTETRIS [25]) integrates SUMO with NS-3 through IP-based sockets and allows implementation of several ITS applications in various programming languages. Very recently, Songchitruksa et. al. developed a closed-loop CV simulator (CONVAS [5,42]) by interlinking VISSIM and NS-3. VNetIntSim [6], another recent closed-loop simulator, couples OPNET and INTEGRATION. A comprehensive survey of contemporary CV simulators can be found in one of our earlier papers [43].

2) Flexible Pair Coupling
In contrast to existing fixed simulator couplings, the VSimRTI [76,77,78] simulation platform allows the flexible integration of traffic, communication, and emission simulators. Therefore, the high modularity of VSimRTI enables the coupling of the most appropriate simulators for a realistic representation of transportation, emissions, communication, driver behavior, and mobility modeling. Based on the requirements of a particular simulation scenario, the most relevant simulators can be chosen. In addition, VSimRTI extends the simulation of electric vehicles. VSimRTI utilizes an ambassador concept to couple arbitrary simulation systems with a remote-control interface. In order to attach an additional simulator, the ambassador interface needs to be implemented. Currently, VSimRTI already includes an interface for traffic simulators SUMO and PHABMACS and the communication simulators ns-3 and OMNeT++.
Below we have summarized all these simulators in the Table 1: Unfortunately, none of these closed-loop simulators described above integrate with hardware-in-the-loop simulation technique. These tools also lack in providing support for simulating large-scale transportation scenario using parallel and distributed computing. Another major limitation is that there is no mechanism available for collecting roadway sensor data from individual intersections and feeding them to the simulation environment to facilitate real-time traffic decision support at the TMCs.

III. PARALLEL SIMULATION OF CONNECTED VEHICLE APPLICATIONS
Need for Parallelism The augmented scale and complexity of urban transportation networks have significantly increased the execution time and resource requirements of vehicular network simulations, exceeding the capabilities of conventional simulators. The need for a parallel and distributed CV simulation environment is inevitable from a smart city perspective where the entire citywide information system will be integrated with numerous services and ITS applications, particularly when the metro-wide multimodal transportation systems get connected to the smart city infrastructure through DSRC.
Unfortunately, contemporary simulation tools do not provide any mechanism for parallel or distributed simulation of CV applications for large-scale transportation networks. Due to the complexity of implementation, earlier attempts [46,47] of parallelizing vehicular network simulation were limited to only open-loop simulators, which do not facilitate real-time feedback control between the transportation and communication parameters. To address the need for ITS practitioners and researchers, we develop a framework for parallelizing closed-loop CV simulation, with the option to incorporate real-time roadway sensor data through hardwarein-the-loop simulation.

Implementation Challenges and Issues
In this section, we identify some of the challenges and issues associated with implementing a parallel closed loop simulator for the large-scale transportation network management. Later we provide insights into the solution approaches that can address these problems.

1) Partitioning of Bi-layer Complex Transportation Network
The fundamental research problem involved in this parallel simulator design is to determine a near-optimal partitioning heuristic using a bi-layer network model-a static road network overlaid with a dynamic vehicular network-connected by the CV technology which spans across both the networks ( Figure  1). Earlier research mainly focused on partitioning static road

CV Technology (Generating Data Traffic)
OBUs RSUs networks for distributed simulation without considering the data traffic generated by DSRC communications. The bi-layer model will address partitioning issues in both the dynamic vehicular network involving CVs (OBUs) as well as the static infrastructure (RSUs) and the interactions between the two levels. The solution approaches in section IV will provide guidelines to incorporate real-world roadway traffic parameters with the data traffic parameters within the partitioning heuristic for connected vehicle environments.
The biggest challenge in partitioning vehicular networks is that the partitions cannot be fully separated. In fact, due to the communication and high mobility, partitions have a high level of interdependency and interactivity (i.e., a message or a vehicle moves from one partition to another) that necessitate communications between partitions to achieve consistency and accuracy. Inefficient partitioning of such networks can produce high communication volume between partitions and high processing overhead within a partition, consequently resulting in low simulation speeds. So, it is necessary to create partitions in such a way that reduces the interactivity and interdependence between them. Another proven NP-hard problem is the loadbalancing problem. Due to the interdependency between events in different partitions, the simulation must be synchronized between the partitions; i.e., low-load (high-speed) partitions must wait for high-load (low-speed) ones to finish. This means that the maximum overall simulation speed is limited to the minimum speed among all the partitions.

2) Reducing Inter-Simulator Communication Overhead
A major problem for parallelizing a closed-loop CV simulator is that it not only requires decomposing the two standalone simulators (the transportation and communication simulators) and synchronizing the components within each simulator, but it also requires a tight synchronization between the two simulators. DSRC technology requires that the vehicles broadcast their current locations every 0.1 seconds, meaning that the two simulators must synchronize ten times per second. This synchronization process adds extra overhead if the two simulators are running on separate computing nodes in a distributed computing environment requiring them to communicate over the Message Passing Interface (MPI). With a shared memory interface between the two simulators running in the same partition, this Inter-Simulator Communication overhead is expected to be reduced. However, using shared memory also creates a race condition between multiple processes running on the same computing node. Hence, there is always a trade-off between contention (shared memory) and latency (distributed memory), which is a major research problem. In addition, in a CV environment, the closed-loop interactions between communication and transportation systems must be executed in real-time to accurately model the impact of one system on its counterpart. For instance, the realtime interactions between SUMO and OMNET++ should facilitate dynamic speed control for the vehicles in the vicinity of traffic signals, where vehicles and signal controllers can exchange information to compute the optimal signal timing and vehicle trajectory.
3) The existence of heterogeneous vehicles Another challenging aspect of simulating transportation network involving CVs is due to the slow market penetration rate of connected vehicles, which implies that during the transition period there will always be two types of vehicles on the road-one that is connected through DSRC (CV) and the other that is not connected (non-CV). It is expected that CV technologies will penetrate the market slowly over the next few years. Hence, until the time comes when all the cars on the road are equipped with factory-built or after-market DSRC devices, there will always be two types of vehicles on the road: one that has DSRC onboard unit (OBU) and the other that does not have OBU. CVs broadcast their actual GPS positions and speed every 0.1 seconds through the basic safety messages (BSMs). So, CVs can be easily identified through the BSM packets. However, detection of non-CV vehicles may not be easy, although it is possible to detect the non-CVs and reconstruct their mobility traces using several techniques. First, we can equip optical sensors to CVs so that the CVs can collect surrounding vehicles' information using the optical sensors. A CV can detect duplicate vehicles by communicating with other CVs. Second, we can reconstruct the mobility traces using some established techniques discussed in [39], [40], and [41]. Third, the data from the loop detector and traffic cameras can further refine non-CV detection and mobility trace reconstruction. At present, there is no closed loop simulator that supports the simulation of CV applications with heterogeneous types of vehicles.
Using the state-of-the-art closed-loop simulator with the support for hardware-in-the-loop simulation, vehicles with attached OBUs will be able to participate in the network-wide communication, while the vehicles without OBUs will not be detected by the simulator. Incorporating non-CVs in the hardware-in-the-loop simulation is quite challenging because the closed-loop CV simulator needs to be fed from several sources of sensor data such as CV traces through BSM messages and non-CV traces from roadway sensors (loop detectors and video detectors). From the input of these sources, the simulator needs to generate realistic mobility traces for the non-CVs, in addition to mapping the actual positions of the CVs where the simulator should graphically represent the CVs and non-CVs differently to distinguish between the actual position and speed vs. projected position and speed.

4) Synchronization problem
Simulation of data traffic is computationally more resource intensive than the simulation of vehicular traffic [6]. This makes the closed-loop simulation of CVs challenging because the imbalance of computational resource requirement causes synchronization problems between the transportation simulator and communication simulator. The synchronization problems happen due to the huge amount of DSRC basic safety messages (BSM) disseminated from each vehicle in every tenth of a second, where each BSM message needs to go through several layers of encapsulation and de-capsulation steps within the wireless network's protocol stack at both ends. Some of the services in the data communication protocols, e.g. error detection, routing, and connection establishment, are computationally more expensive compared to the services from the vehicular traffic simulator that do not require passing through multiple layers of protocols. In fact, simulation of vehicular traffic only involves trace generation using microscopic mobility models. Hence, the data traffic simulator primarily causes the bottleneck. Typically, the data traffic is simulated using network simulators such as OPNET, OMNET++, Qualnet, NS-2, or NS-3. One experiment in [29] demonstrated that the simulation of a 200-node network for only one minute generated more than 4,600,000 events and required 16 minutes of CPU time. The increasing complexity of the protocol stacks in end devices further aggravates this problem and has spurred efforts to develop parallel network simulators.

5) Scalability of Parallel Simulation
The scalability of parallel systems depends on the ratio of time spent in computation vs. communication. For any parallel system, the fraction of time spent in inter-process communication increases with the number of processors while the fraction of time spent in actual computation decreases. Initially, the computational time is greater than the communication time for a lower number of processors. The computational time decreases with respect to the communication time with the increase of the number of processors. At some point, for a specific number of processors (p), the communication time starts dominating over computation time. This value of p essentially determines how better the system is scalable. The higher the value of p, the better the scalability. Therefore, scalability is one of the most important problems in any parallel system, especially when it involves both distributed and shared memory architecture. Hence the architecture of such hybrid parallel system needs to be designed in such a way that reduces the inter-process communication (IPC) overhead and increases the scalability. It is noteworthy to mention that this IPC could take place between the transportation simulator and network simulator using shared memory (using OpenMP) or between the instances of the same simulator running different partitions on distributed cluster nodes (through MPI). Without achieving a certain level of scalability, the system will not be able to simulate a city-wide scenario with several hundred thousand vehicles and millions of BSM messages in every minute. Ahmed et. al. studied the scalability issues in terms of memory usage and execution time using VNetIntSim [33,34] proving that the number of wireless nodes (vehicles) and the data traffic rate per vehicle are the primary reasons behind the scalability issue. In this section, we present a conceptual framework of an Integrated Distributed Connected Vehicle Simulator (IDCVS) and in section V we discuss the technical approaches for implementing such a robust simulation tool. Figure 2 below shows our hypothetical model of IDCVS system that includes hardware-in-the-loop simulation techniques for both CVs and non-CVs. IDCVS will incorporate two basic modules-a CLOsed-loop Parallel Simulator (CLOPS) and a Hardware-In-the-Loop Simulation (HILS) module.

Hardware-In-the-Loop Simulation (HILS)
HILS will have an interface to receive the sensor data from both CVs and non-CVs through multiple sources. For non-CVs, the approximate location and speed can be detected through video detectors and inductive loop detectors, and this information will be passed as input to the HILS module. We can use the video detection and the loop detector software that can supply the sensor data to the HILS receiver component. On the other hand, the CV's can be detected easily through the BSM messages received by the RSUs. Once the sensor data is received, additional data-preprocessing, filtering, and extrapolation will be needed before the data can be used by CLOPS. This will require developing filtering algorithms for loop-detection and video-detection data to isolate the CV traces from the non-CV traces.

CLOsed-loop Parallel Simulator (CLOPS)
A CLOsed-loop Parallel Simulator (CLOPS) can be developed by interlinking OMNET++ with SUMO where both simulators are open source. An efficient partitioning heuristic will decompose the complex transportation network into two separate sets of partitions-where each set of the partition will be sent to the individual simulator (SUMO and OMNET++). It might appear that CLOPS could be developed as a parallel and distributed framework on top of Veins since Veins also utilizes a coupling between SUMO and OMNET++. However, since Veins does not support heterogeneous vehicles, it is not possible to extend Veins for the simulation scenarios involving both CVs and non-CVs. In addition, CLOPS may have the capability to vary the ratio of CVs to the non-CVs as per the market penetration rate. This requires a non-uniform partitioning between SUMO and OMNET++.

Modes of Operation for IDCVS
One important feature of this conceptual IDCVS system is that it will have the option to simulate in two different modesclosed-loop simulation (CLSim) mode and HILS-mode. The CLSim mode will simulate without sensor data, in this case the entire simulation will be run within CLOPS. To incorporate both DSRC-equipped and non-equipped vehicles on a CLSim scenario, we can randomly distribute the CV vehicles within the road network based on a user-specified technology penetration rate. On the other hand, the HILS-mode will enable simulation based on real-time sensor data.

V. IMPLEMENTATION APPROACHES
In this section, we discuss the possible technical approaches to address the challenges pertaining to implementation of the integrated simulator.

Developing Network Partitioning Heuristic
A crucial challenge for the partitioning problem described in section III is that, due to the imbalance of computational resource requirements between transportation simulator (SUMO) and network simulator (OMNET++), a single partitioning scheme may not work for both of the simulators. Apart from that, the number of vehicles will also vary among the two simulators where SUMO needs to simulate the traces for all vehicles (both CVs and non-CVs) whereas OMNET++ only simulates data traffic generated from the CVs. If a single partitioning heuristic is used, the synchronization problem will be further aggravated. Hence, it is necessary to have two separate partitioning schemes for SUMO and OMNET++.
A Boolean matrix based vehicular network partitioning algorithm has been developed by Hoque et. al [44,45] which is expected to be incorporated into our distributed simulation architecture involving hybrid parallelization model. In our recent work related to road-network partitioning [30,31], we have identified the following issues and parameters that play vital roles in designing an efficient partitioning heuristic:

1) System boundary nodes of each partition:
The total number of inter-process communication or messaging depends on the number of system boundary nodes of each partition.
2) The number of partitions: Almost every graph partitioning algorithm is based on a pre-specified number of partitions, which may not always generate the optimal solution in practice. Instead of specifying an exact number of partitions, an upper bound and lower bound can be provided as input to the algorithm to determine the best partitioning solution within the specified range.

3) Intersection cut:
If an intersection is considered as a boundary node for a partition, then a significant amount of vehicle mobility data must be communicated between the partitions. In this context, an important factor-whether to prioritize signalized intersection over un-signalized intersection as a candidate for boundary node-remains open for further research, which should be investigated.

4) Link/Edge cut:
When a link or edge is selected to be cut then the traffic volume along the cut link is directly proportional to the amount of information exchanged between the two partitions along the link. In this case, a good strategy would be to cut the links with minimum traffic to reduce the communication overhead between partitions. Partitioning Approach for SUMO To create the network graph, the OSM file of the experimental city can be downloaded from the www.openstreet.org website. To avoid unnecessary complexities, residential street, service path, footway, cycleway, motorway, and unclassified roads can be excluded from the graph. Table 2 shows some suggested parameters that can be incorporated to generate the weighted graph matrix. It could be easily possible to extend an existing graph partitioning software like METIS [37] for generating the partitions of the transportation network. Figure 3 shows our preliminary transportation network partitioning using the above-mentioned parameters with the help of METIS. METIS is a very stable partitioning package implementing the popular Kernighan-Lin heuristic. METIS divides a graph into three phases: coarsening, partitioning, and uncoarsening. In coarsening phase, the heavy edge matching scheme can be used, whereas, in the uncoarsening phase, the Kernighan-Lin graph refinement algorithm can be used. The coarsest graph can be bisected using graph growing followed by boundary Kernighan-Lin algorithm with graph partitioning using recursive bisection technique. The input for METIS can be provided using the generated graph matrix and weight parameters. Node weight All signalized intersections in the OSM data will be identified. These types of intersections or nodes will be assigned a higher weight. Un-signalized intersections will be assigned the sum of the number of incoming and outgoing lanes as the weight.

Link length
The length between two nodes will be calculated using the Haversine formula:

Number of lanes
The number of lanes of a road segment or a link will be extracted from the OSM data.
Link density The density of a road segment or link will be extracted from the Google Map Application's newly introduced traffic layer [36]. The density is expressed in three categories: low, medium, and high.
Link priority The road segment will be assigned a priority index based on the weighted summation of link length, the number of lanes, and link density.
Partitioning Approach for OMNET++ Some of the key factors concerning partitions for OMNET++ in the context of closed-loop parallel simulation for reduced interactivity and interdependence include vehicle mobility, communication events and external stimuli from the simulated transportation applications. These factors directly influence the previously mentioned challenges pertaining to network partitioning. The application stimuli are the drive for CV communications, which can be sporadic or proactive. The transportation network information such as the road network (road links, road nodes), vehicle density on each link, and vehicle speeds and distribution determine the vehicular mobility. This information can be further utilized to quantify the number of communication events. The approach to optimize partitions is to consider the number of discrete events in the communication network as the basis for drawing the boundary between the connected components. For example, one way to incorporate this approach is to employ the vehicle density and the length of each link as link weights in partitioning techniques (such as the minimum cut or minimum k-cut algorithms) to partition the network and minimize the interactivity between different portions. The lower the density and the longer the length of a link, the higher the possibility that the link is a cut link in the network. The rationale is that the density and length represent the continuity of the communication route on this link. Therefore, the lower this ratio (density/length), the less communication between the ends of the link. In addition, the partitions need to be adaptive to the dynamics of the application stimuli and the mobility. To address this issue, we can consider the simulation granularity and duration of the current partition time. The goal is to develop an intelligent algorithm to schedule the partitioning job.

Design of Closed-loop Parallel Simulator (CLOPS)
The closed-loop parallel simulator (CLOPS) integrates SUMO and OMNET++ as two standalone simulators. OMNET++ has the flexibility to dynamically create and delete nodes; this capability is necessary for a parallel simulation environment since the wireless vehicular nodes will be distributed in multiple network partitions based on geographic location. In addition, OMNET++ provides support for both distributed and shared memory computing which is needed for this project. The PHY and MAC layers of DSRC (IEEE 802.11p and IEEE 1609.4) have already been implemented in the OMNET++ platform by the open-source research community, which can be utilized in our research. This is a big advantage compared with OPNET since OPNET does not currently include the DSRC protocol stack. CLOPS will incorporate hybrid parallelization schemes for both the traffic simulator and network simulator. This hybrid parallelization schemes will allow the integrated platform to run in parallel on clusters of computers within a supercomputing facility. The hybrid inter-process communication will be incorporated using MPI and OpenMP. In our preliminary work in [32], we investigated the impact of communication overhead on the overall simulation time and implemented a new partialresult accumulation pattern for reducing inter-process communication overhead. Figure 4 illustrates the architecture of the envisioned parallel system that incorporates hybrid parallelism. Both transportation and communication simulators will have master controllers (the Transportation Simulation Controller (TSC) and Network Simulation Controller (NSC)) that will coordinate the computational load distribution among the parallel sub-processes. Each of these sub-processes is supposed to simulate a portion of the transportation network defined by the network partitioning scheme. The controller will communicate with the sub-processes using MPI, while a transportation simulator sub-process corresponding to a specific partition communicates with its network simulator counterpart using OpenMP. It would be beneficial to utilize two levels of parallelization: network and event levels. At the network level, the overall network can be divided into multiple partitions for both SUMO and OMNET++, each of which will run on a different machine. The TSC and NSC are responsible for managing the loads and synchronizing the partitions within the transportation and communication domains, respectively. At the event level, events can run in parallel within a pre-calculated look-ahead interval. The calculation of the optimum look-ahead interval is crucial in the event-level parallelization. In fact, the look-ahead interval involves a tradeoff between the simulation speed and output accuracy. In the event-level parallelization, utilizing parameters such as node locations and the number of hops between two nodes can increase the scalability of the parallel simulation. For instance, nodes that are spatially separated by long distances can run events in parallel within longer look-ahead intervals without affecting the output.
The communication between the TSC and NSC can be achieved by using shared memory. The vehicles' locations will be calculated and sent to the NSC periodically through the shared memory, and any required application information between the TSC and NSC will be exchanged through the shared memory. Compared to TCP/IP message passing, shared memory has the advantages of reliability and the highest possible speed of information exchange. In contrast, the message size in TCP/IP message passing is limited; thus, in the case of large network size, a large number of messages are needed for each location update. Consequently, TCP/IP message passing may create a communication bottleneck, resulting in the degradation of simulation speed.

Incorporating Hardware-In-the-Loop Simulation (HILS)
The benefits of incorporating hardware-in-the-loop simulation in the parallel simulation framework are twofold: (a) real-time visualization of system-wide traffic and (b) fidelity testing of V2I applications. They are described as follows.

a) Visualization:
One of the purposes of integrating hardware-in-the-simulation (HILS) is to provide real-time traffic information obtained through various roadway sensors to the ITS practitioners monitoring roadway conditions from TMC (traffic management center). The HILS module aggregates roadway sensor data to facilitate visualization of system-wide traffic. A potential use case for our HILS-incorporated parallel CV simulation tool is to provide a realistic prediction of the consequences of traffic change-such as transit bus delays or tentative queue length considering the preemptive detour advisory disseminated through DSRC-enabling a TMC official to make an informed decision when a major corridor needs to undergo closure of lanes due to maintenance.

b) Fidelity Testing:
Another important benefit of integrating HILS is to validate the simulation results of V2I applications. For example, eco intersection approach applications [48-60, 79,80] using signal phase and timing (SPaT) information could be first simulated in the CLOsed-loop Parallel Simulator (CLOPS) and then validated using actual roadway sensor data through HILS.
To capture the movement of the non-CVs, several types of detectors can be used such as the magnetometer, inductive loop detection (ILD), video detection, etc. Loop detection is also capable of counting traffic. But it is not 100% reliable for actual traffic counts because the loops in the adjacent through lanes are often tied together for one output for the movement. To solve this problem, the latest video detection technology capable of counting actual traffic can act as a complement for the loop detector. Figures 5(a) and 5(b) shows how the two software detect vehicles at the intersection through software.
Since the target is to simulate both CVs and non-CVs, it is necessary to feed the vehicles' information to the traffic and communication simulators. The RSU can automatically detect the CVs from the BSM packets, but the loop detection and video detection techniques are necessary for detecting the non-CVs. Once the RSU gets the data from all the sources (e.g. BSM packets, inductive loop, video, and magnetometer), a filtering algorithm separates the non-CVs from the CVs using the BSM packets. There is a re-identification issue regarding the use of loop detector, video detection, and magnetometer. In order to eliminate the re-identification problem, we will consolidate the roadway sensor data from three sources along with CV data to accurately identify both CVs and non-CVs. However, detection of the non-CV is not sufficient for the hardware in loop simulation. The mobility trace of a non-CV between two intersections is needed. A car-following model between one/two CVs and a non-CV can be used to extrapolate the missing trace of a non-CV vehicle. For example, the missing mobility trace of a non-CV vehicle can be extrapolated using two CVs' mobility traces where one CV precedes the non-CV and one CV that follows the non-CV. More generally, we can utilize the existing stochastic or probabilistic traffic flow and mobility trace reconstruction techniques [39 -41] to reconstruct the mobility traces of the non-CV vehicles considering the CV vehicles as the probe vehicles (details discussed in the second challenge of HILS). Other sensors in an intersection can aid to refine the reconstruction. Some of the challenges associated with integrating hardwarein-the-loop simulation are described below:

1) Isolating CV traces from loop-detection and video detection data
The hardware-in-the-loop simulation (HILS) technique can capture roadway sensor data from four different sources-DSRC broadcast messages, inductive loops, video detectors, and wireless magnetometers. Unfortunately, the roadway sensors at intersections cannot differentiate between a CV and non-CV. So, a reliable filtering mechanism is needed to identify the CVs among all the traffic by filtering out the CV data from other two sensors' data based on the GPS position and loop detection timings.

1) Missing traces between two intersections
It is very challenging to emulate non-CVs based on sensor data because of the missing traces between two intersections since they can only be detected at the intersections [81][82][83]. Also, SUMO generated mobility traces between two intersections are the only sources to fill up the missing traces. However, this approach may give some margin of error since some vehicles may arrive at their destination before reaching the next intersection while some other vehicles may start from a midpoint between the two intersections. Since the goal is to approximate the expected traffic between two intersections at a given time, some established statistical models are necessary to validate the simulation results between two intersections. For example, the modal activity-based probabilistic model [39] can be utilized to reconstruct missing non-CV traces using the CV traces. The modal activity sequence in the probabilistic model [39] utilizes the repetitive stop-and-go behaviors of vehicles between two intersections which in fact result in a particular pattern of the activities such as idling, acceleration, cruising, and deceleration. The model generates several vehicle dynamic states using the pattern, ranks the dynamic states using probability distribution functions, and reconstructs the second by second trajectories from the dynamic states. It might be beneficial to incorporate the traffic flow reconstruction methods described in [40] and [41] where each method discusses the traffic flow reconstruction technique using the traffic data such as speeds, positions, and timestamps collected by probe vehicles. In our model, we plan to reconstruct the traces of non-CV vehicles considering the CV vehicles as the probe vehicles. The trajectories of CVs will be reflected in the simulation with almost 100% accuracy given the precision level of GPS sensors integrated with DSRC OBUs. On the other hand, the trajectories of the non-CVs can be deterministically traced near the intersections with the aid of roadway sensors. This means that the non-CVs (real) will be transformed into virtual vehicles between two intersections, while CVs will continue to provide actual mobility traces even within the road segment between two intersections. Between two intersections, the traces of non-CVs can be reconstructed up to a certain level of accuracy by utilizing relative distance information obtained through the sensors equipped in CVs. Even if there is no CV available within the vicinity of a virtual non-CV, still the recent microscopic car-following models can generate a 'close-toreal' trajectory between two intersections in SUMO.
2) Inaccurate traffic count by loop detectors Loop detection can detect traffic but is less reliable for actual traffic counts because the loops in the adjacent through lanes are often tied together for one output for the movement. Also, due to the length of the loop (40 to 50 feet) at the stop bar, multiple vehicles may fall over the same loop or the loops tied together at the same time which reduces vehicle count accuracy. The current reported accuracy from trace reconstruction model is about 80% [40,41]. We believe that once we consolidate the sensor data from three different sources and cross match with CV data, the accuracy will be increased. However, some complex vehicle identification algorithm has to be developed to accomplish this goal. The improvement of accuracy level from roadway sensor data is left as an open research issue for the community.
3) Different data formats Typically, data loggers' records include events at an intersection, including a light turning green, a light turning yellow, a vehicle detector turning on, a vehicle detector turning off, and pedestrian walk phase active. While CV data follows DSRC beacon format, loop detector, and video detector inputs are again in a different format. Thus, different pre-processing algorithms are needed.

CONCLUSION
In this paper, we have discussed a conceptual model that can simulate system-wide changes in traffic parameters on roadways involving both connected vehicles and regular vehicles. We have identified the major challenges and issues for implementing the hardware-in-the-loop simulation and incorporating hybrid parallelism in the closed-loop simulation. We have also discussed the solution approaches for the challenges and issues involved in implementing the conceptual model. However, only a few solutions have actually been implemented. We have discussed possible technical approaches to address the challenges and implementation issues. Our ongoing efforts are directed towards implementation of this model and evaluation of the scalability for emulating metrowide transportation network. It is noteworthy to mention that a closed-loop integrated parallel simulator may still lack some realistic features within individual simulator modules (e.g. SUMO and OMNET++). However, given that both of the standalone simulators (SUMO and OMNET++) are opensource in nature, it is possible to add the missing features or refine the parameters within each of the simulators (SUMO and OMNET++) even after the implementation of the closed-loop simulator. Our focus in this paper is to point out the challenges of designing the interface between two simulators in the parallel platform, considering the traffic simulator and network simulator as a "black box".


extfieldsofstudy:
Computer Science
Mathematics

text:
Classifier-agnostic saliency map extraction

Extracting saliency maps, which indicate parts of the image important to classification, requires many tricks to achieve satisfactory performance when using classifier-dependent methods. Instead, we propose classifier-agnostic saliency map extraction, which finds all parts of the image that any classifier could use, not just one given in advance. We observe that the proposed approach extracts higher quality saliency maps and outperforms existing weakly-supervised localization techniques, setting the new state of the art result on the ImageNet dataset. We made our code publicly available at https://github.com/kondiz/casme .

INTRODUCTION
The success of deep convolutional networks for large-scale object recognition Krizhevsky et al. (2012); Simonyan & Zisserman (2014); Szegedy et al. (2015); He et al. (2016) has spurred interest in utilizing them to automatically detect and localize objects in natural images. Pioneering this direction, Simonyan et al. (2013) and Springenberg et al. (2014) demonstrated that the gradient of the class-specific score of a given classifier could be used for extracting a saliency map of an image. Such classifier-dependent saliency maps can be utilized to analyze the inner workings of a specific network. However, as only the part of the image that is used by a given model is highlighted, these methods are not identifying all "evidence" in a given image. They also tend to be noisy, covering many irrelevant pixels and missing many relevant ones. Therefore, much of the recent work has focused on introducing regularization techniques of correcting such classifier-dependent saliency maps. For instance, Selvaraju et al. (2017) propose averaging multiple saliency maps created for perturbed images to obtain a smooth saliency map. We argue, however, that applying tricks and tweaks on top of methods that were designed to analyze inner workings of a given classifier is not a principled way to get saliency maps that focus on all useful evidence.
In this work, we aim to find saliency maps indicating pixels which aid classification, i.e. we want to find pixels in the input image such that if they were masked, it would confuse an unknown classifier. Assuming we were given a classifier, a naive approximate solution would be to train a generative model to output a mask (a saliency map) confusing that classifier. That can be achieved using a simple GAN-like approach (Goodfellow et al., 2014) where the classifier acts as a fixed discriminator. Unfortunately, as we prove experimentally, this solution suffers from the same issues as prior approaches. We argue that the strong dependence on a given classifier lies at the center of the problem. To tackle this directly we propose to train a saliency mapping that is not strongly coupled with any specific classifier. Our approach, a class-agnostic saliency map extraction, can be formulated as a practical algorithm that realizes our goal.
Our focus on classifier-agnostic saliency maps is not our objective per se, it is a remedy that resolves the core problem. The proposed approach results in a neural network based saliency mapping that only depends on an input image. We qualitatively find that it extracts higher quality saliency maps compared to classifier-dependent methods, as can be seen in Fig. 2. Extracted saliency maps show all the evidence without using any symptom-masking methods: difficult to tune regularization penalties (such as total variation), exotic activation functions, complex training procedures or image preprocessing tricks (such as superpixels), etc. We also evaluate our method quantitatively by using the extracted saliency maps for object localization. We observe that the proposed approach outperforms the existing weakly-supervised techniques setting the new state of the art result on the ImageNet dataset and closely approaches the localization performance of a strongly supervised model. Furthermore, we experimentally validate that the proposed approach works reasonably well even for classes unseen during training.
Our method has many potential applications, in which being classifier-agnostic is of primary importance. For instance, in medical image analysis, where we are interested not only in class prediction but also in indicating which part of the image is important to classification. Importantly, however, it is criticial to indicate all parts of the image, which can influence diagnosis, not just ones used by a specific classifier.

CLASSIFIER-AGNOSTIC SALIENCY MAP EXTRACTION
In this paper, we tackle a problem of extracting a salient region of an input image as a problem of extracting a mapping m : . Such a mapping should retain (= 1) any pixel of the input image if it aids classification, while it should mask (= 0) any other pixel.

CLASSIFIER-DEPENDENT SALIENCY MAP EXTRACTION
Earlier work has largely focused on a setting in which a classifier f was given (Fong & Vedaldi, 2017;Dabkowski & Gal, 2017). These approaches can be implemented as solving the following maximization problem: where S is a score function corresponding to a classification loss. That is, where denotes elementwise multiplication (masking), R(m) is a regularization term and l is a classification loss, such as cross-entropy. We are given a training set D = {(x 1 , y 1 ), . . . , (x N , y N )}. This optimization procedure could be interpreted as finding a mapping m that maximally confuses a given classifier f . We refer to it as a classifier-dependent saliency map extraction.
A mapping m obtained with a classifier f may differ from a mapping m found using f , even if both classifiers are equally good in respect to a classification loss for both original and masked images, i.e.
This property is against our definition of the mapping m above, which stated that any pixel which helps classification should be indicated by the mask (a saliency map) with 1. The reason why this is possible is these two equally good, but distinct classifiers may use different subsets of input pixels to perform classification.
An example This behaviour can be intuitively explained with a simple example, illustrating an extreme special case. Let us consider a data set in which all instances consist of two identical copies of images concatenated together, that is, for all x n , x n = [x n ; x n ], where x n ∈ R W/2×H×3 . For such a data set, there exist at least two classifiers, f and f , with the same classification loss. The classifier f uses only the left half of the image, while f uses the other half. Each of the corresponding mappings, m and m , would then indicate a region of interest only on the corresponding half of the image. When the input image does not consist of two concatenated copies of the same image, it is unlikely that two equally good classifiers will use disjoint sets of input pixels. Our example is to show an extreme case when it is possible.

CLASSIFIER-AGNOSTIC SALIENCY MAP EXTRACTION
In order to address the issue of saliency mapping's dependence on a single classifier, we propose to alter the objective function in Eq.
(1) to consider not only a single fixed classifier but all possible classifiers weighted by their posterior probabilities. That is, where the posterior probability, p(f |D, m ), is defined to be proportional to the exponentiated classification loss L, i.e., p(f |D, m ) ∝ p(f ) exp(−L(m , f )). Solving this optimization problem is equivalent to searching over the space of all possible classifiers, and finding a mapping m that works with all of them. As we parameterize f as a convolutional network (with parameters denoted as θ f ), the space of all possible classifiers is isomorphic to the space of its parameters. The proposed approach considers all the classifiers and we call it a classifier-agnostic saliency map extraction.
In the case of the simple example above, where each image contains two copies of a smaller image, both f and f , which respectively look at one and the other half of an image, the posterior probabilities of these two classifiers would be the same 1 . Solving Eq. (4) implies that a mapping m must minimize the loss S for both of these classifiers.

ALGORITHM
Algorithm 1: Classifier-agnostic saliency map extraction input :an initial classifier f (0) , an initial mapping m (0) , dataset D, number of iterations K output :the final mapping m (K) Initialize a sample set The optimization problem in Eq. (4) is, unfortunately, generally intractable. This arises from the intractable expectation over the posterior distribution. Furthermore, the expectation is inside the optimization loop for the mapping m, making it even harder to solve.
Thus, we approximately solve this problem by simultaneously estimating the mapping m and the expected objective. First, we sample one f (k) with the posterior probability p(f |D, m (k−1) ) by taking a single step of stochastic gradient descent (SGD) on the classification loss with respect to θ f (classifier f parameters) with a small step size: This is motivated by earlier work (Welling & Teh, 2011;Mandt et al., 2017) which showed that SGD performs approximate Bayesian posterior inference.
We have up to k + 1 samples 2 from the posterior distribution F (k) = f (0) , . . . , f (k) . We sample 3 f ∈ F (k) to get a single-sample estimate of m in in Eq. (4) by computing S(m (k−1) , f ). Then, we use it to obtain an updated θ m (mapping m parameters) by We alternate between these two steps until m (k) converges (cf. Alg. 1). Note, that our algorithm resembles the training procedure of GANs (Goodfellow et al., 2014), where mapping m takes the role of a generator and the classifier f can be understood as a discriminator.
Score function The score function S(m, f ) estimates the quality of the saliency map extracted by m given a data set and a classifier f . The score function must be designed to balance the precision and recall. The precision refers to the fraction of relevant pixels among those marked by m as relevant, while the recall is the fraction of pixels correctly marked by m as relevant among all the relevant pixels. In order to balance these two, the score function often consists of two terms.
The first term is aiming to ensure that all relevant pixels are included (high recall). As in Eq.
(2), a popular choice has been the classification loss based on an input image x masked out by 1 − m(x). In our preliminary experiments, however, we noticed that this approach leads to obtaining masks with adversarial artifacts. We hence propose to use the entropy H(f ((1 − m(x)) x)) instead. This makes generated masks cover all salient pixels in the input, avoiding masks that may sway the class prediction to a different, but semantically close class. For example, from one dog species to another. The second term, R(m), excludes a trivial solution, m simply outputting an all-ones saliency map, which would achieve maximal recall with low very precision. In order that, we must introduce a regularization term. Some of the popular choices include total variation (Rudin et al., 1992) and L 1 norm. For simplicity, we use the latter only.
In summary, we use the following score function for the class-agnostic saliency map extraction: where λ R > 0 is a regularization coefficient.
Thinning As the algorithm collects a set of classifiers, f (k) 's, from the posterior distribution, we need a strategy to keep a small subset of them. An obvious approach would be to keep all classifiers but this does not scale well with the number of iterations. We propose and empirically evaluate a few strategies. The first three of them assume a fixed size of F (k) . Namely, keeping the first classifier only, denoted by F (F (k) = f (0) ), the last only, denoted by L (F (k) = f (k) ) and the first and last only, denoted by FL (F (k) = f (0) , f (k) ). As an alternative, we also considered a growing set of classifiers where we only keep one every 1000 iterations (denoted by L1000) but whenever |F (k) | = 30, we randomly remove one from the set. Analogously, we experimented with L100.
Classification loss Although we described our approach using the classification loss computed only on masked images, as in Eq. (3), it is not necessary to define the classification loss exactly in this way. In the preliminary experiments, we noticed that the following alternative formulation, inspired by adversarial training (Szegedy et al., 2013), works better: We thus use the loss as defined above in the experiments. We conjecture that it is advantageous over the original one in Eq.
(3), as the additional term prevents the degradation of the classifier's performance on the original, unmasked images while the first term encourages the classifier to collect new pieces of evidence from the images that are masked.

EXPERIMENTAL SETTINGS
Dataset: ImageNet Our models were trained on the official ImageNet training set with ground truth class labels Deng et al. (2009). We evaluate them on the validation set. Depending on the experiment, we use ground truth class or localization labels.

ARCHITECTURES
Classifier f and mapping m We use ResNet-50 (He et al., 2016) as a classifier f in our experiments. We follow an encoder-decoder architecture for constructing a mapping m. The encoder is implemented also as a ResNet-50 so its weights can be shared with the classifier or it can be separate. We experimentally find that sharing is beneficial. The decoder is a deep deconvolutional network that ultimately outputs the mask of an input image. The overall architecture is shown in Fig. 1. Details of the architecture and training procedure are in the appendix.
Regularization coefficient λ R As noticed by Fan et al. (2017), it is not trivial to find an optimal regularization coefficient λ R . They proposed an adaptive strategy which gets rid of the manual selection of λ R . We, however, find it undesirable due to the lack of control on the average size of the saliency map. Instead, we propose to control the average number of relevant pixels by manually setting λ R , while applying the regularization term R(m(x)) only when there is a disagreement between f (x) and f ((1 − m(x)) x). We then set λ R for each experiment such that approximately 50% of pixels in each image are indicated as relevant by a mapping m. In the preliminary experiments, we further noticed that this approach avoids problematic behavior when an image contains small objects, earlier observed by Fong & Vedaldi (2017). We also noticed that the training of mapping m is more stable and effective when we use only images that the classifier is not trivially confused on, i.e. predicts the correct class for the original images.

EVALUATION
In our experiments we only use a single architecture explained in subsection 3.1. We use the abbreviation CASM (classifieragnostic saliency mapping) to denote the final model obtained using the proposed method. Our baseline model (Baseline) is of the same architecture and it is trained with a fixed classifier (classifier-dependent saliency mapping) realized by following thinning strategy F.
Following the previous work (Cao et al., 2015;Fong & Vedaldi, 2017; we discretize our mask by computing where m(x) is the average mask intensity and α is a hyperparameter. We simply set α to 1, hence the average of pixel intensities is the same for the input mask m(x) and the discretized binary mask b(x).
To focus on the most dominant object we take the largest connected component of the binary mask to obtain the binary connected mask.
Visualization We visualize the learned mapping m by inspecting the saliency map of each image in three different ways. First, we visualize the masked-in image b(x) x, which ideally leaves only the relevant pixels visible. Second, we visualize the masked-out image (1 − b(x)) x, which highlights pixels irrelevant to classification. Third, we visualize the inpainted masked-out image using an inpainting algorithm (Telea, 2004). This allows us to inspect whether the object that should be masked out cannot be easily reconstructed from nearby pixels.
Classification by multiple classifiers In order to verify our claim that the proposed approach results in a classifier-agnostic saliency mapping, we evaluate a set of classifers 4 on the validation sets of masked-in images, masked-out images and inpainted masked-out images. If our claim is correct, we expect the inpainted masked-out images created by our method to break these classifiers, while the masked-in images would suffer minimal performance degradation.
Object localization As the saliency map can be used to find the most dominant object in an image, we can evaluate our approach on the task of weakly supervised localization. To do so, we use the ILSVRC'14 localization task. We compute the bounding box of an object as the tightest box that covers the binary connected mask.
We use three metrics to quantify the quality of localization. First, we use the official metric (OM) from the ImageNet localization challenge, which considers the localization successful if at least one ground truth bounding box has IOU with predicted bounding box higher than 0.5 and the class prediction is correct. Since OM is dependent on the classifier, from which we have sought to make our mapping independent, we use another widely used metric, called localization error ( evaluate the original saliency map, of which each mask pixel is a continuous value between 0 and 1, by the continuous F1 score. Precision P and recall R are defined as the following: where B * (x) is the ground truth bounding box. We compute F1 scores against all the ground truth bounding boxes for each image and report the highest one among them as its final score.

RESULTS AND ANALYSIS
Visualization and statistics We randomly select seven consecutive images from the validation set and input them to two instances of CASM (each using a different thinning strategy -L or L100) and Baseline. We visualize the original (clean), masked-in, masked-out and inpainted masked-out images in Fig. 2. The proposed approach produces clearly better saliency maps, while the classifier-dependent approach (Baseline) produces so-called adversarial masks (Dabkowski & Gal, 2017).
We further compute some statistics of the saliency maps generated by CASM and Baseline over the validation set. The masks extracted by CASM exhibit lower total variation (2.5 × 10 3 vs. 7.0 × 10 3 ), Figure 3: The classification accuracy of the ImageNet-trained convolutional networks on masked-in images (left), masked-out images (center) and inpainted masked-out images (right). Orange and blue dots correspond to ResNet-50 models and all the other types of convolutional networks, respectively. We observe that the inpainted masked-out images obtained using Baseline are easier to classify than those using CASM, because Baseline fails to mask out all the relevant pixels, unlike CASM. On the right panel, most of the classifiers evaluated on images with random masks achieve accuracy higher than 40% and are not shown. We add jitters in the x-axis to make each dot more distinct from the others visibly.
indicating that CASM produced more regular masks, despite the lack of explicit TV regularization. The entropy of mask pixel intensities is much smaller for CASM (0.05 vs. 0.21), indicating that the mask intensities are closer to either 0 or 1 on average. Furthermore, the standard deviation of the masked out volume is larger with CASM (0.19 vs. 0.14), indicating that CASM is capable of producing saliency maps of varying sizes dependent on the input images.
Classification As shown on the left panel of Figure 3, the entire set of classifiers suffers less from the masked-in images produced by CASM than those by Baseline. We, however, notice that most of the classifiers fail to classify the masked-out images produced by Baseline, which we conjecture is due to the adversarial nature of the saliency maps produced by Baseline approach. This is confirmed by the right panel which shows that simple inpainting of the masked-out images dramatically increases the accuracy when the saliency maps were produced by Baseline. The inpainted masked-out images by CASM, on the other hand, do not benefit from inpainting, because it truly does not maintain any useful evidence for classification.
Localization We report the localization performance of CASM, Baseline and prior works in Table 1 using two different metrics. Most of the existing approaches, except for Fan et al. (2017), assume the knowledge of the target class, unlike our work. CASM performs better than all prior approaches including the classifier-dependent Baseline. The difference is statistically significant. For ten separate training runs with random initialization the worst scores 36.3 and the best 36.0 with the average of 36.1. The fully supervised approach is the only approach that outperforms CASM.
Thinning strategies In Table 2 (a-e), we compare the five thinning strategies described earlier, where F is equivalent to the Baseline. According to LE and OM metrics, the strategies L100 and L1000 perform better than the others, closely followed by L. These three strategies also perform the best in term of F1.
Sharing the encoder and classifier As clear from Fig. 1, it is not necessary to share the parameters of the encoder and classifier. Our experiments, however, reveal that it is always beneficial to share them as shown in Table 2.
Score function Unlike Fan et al. (2017), we use separate score functions for training the classifier and the saliency mapping. We empirically observe in Table 2 that the proposed use of entropy as a score function results in a better mapping in term of OM and LE. The gap, however, narrows as we use better thinning strategies. On the other hand, the classification loss is better for F1 as it makes CASM focus on the dominant object only. Because we take the highest score for each ground truth bounding box, concentrating on the dominant object yields higher scores.

UNSEEN CLASSES
Since the proposed approach does not require knowing the class of the object to be localized, we can use it with images that contain objects of classes that were not seen during training neither by the classifier f (k) nor the mapping m. We explicitly test this capability by training five different CASMs on five subsets of the original training set of ImageNet.
We first divide the 1000 classes into five disjoint subsets (denoted as A, B, C, D, E and F) of sizes 50, 50, 100, 300, 300 and 200, respectively. We train our models (in all stages) on 95% images (classes in B, C, D, E and F), 90% images (classes in C, D, E and F), 80% images (classes in D, E and F), 50% images (classes in E and F) and finally on 20% of images only (classes in F only). Then, we test each saliency mapping on all the six subsets of classes independently. We use the thinning strategy L for computational efficiency in each case.
All models generalize well and the difference between their accuracy on seen or unseen classes is negligible (with exemption of the model trained on 20% of classes). The general performance is a little poorer which can be explained by the smaller training set. In Table 3, we see that the proposed approach works well even for localizing objects from previously unseen classes. The gap in the localization error between the seen and unseen classes grows as the training set shrinks. However, with a reasonably sized training set, the difference between the seen and unseen classes is small. This is an encouraging sign for the proposed model as a class-agnostic saliency map.

RELATED WORK
The adversarial localization network Fan et al. (2017) is perhaps the most closely related to our work. Similarly to ours, they simultaneously train the classifier and the saliency mapping which does not require the object's class at test time. There are four major differences between that work and ours. First, we use the entropy as a score function for training the mapping, whereas they used the classification loss. This results in obtaining better saliency maps as we have shown earlier. Second, we make the training procedure faster thanks to tying the weights of the encoder and the classifier, which also results in a much better performance. Third, we do not let the classifier shift to the distribution of masked-out images by continuing training it on both clean and masked-out images. Finally, their mapping relies on superpixels to build more contiguous masks which may miss small details due to inaccurate segmentation and makes the entire procedure more complex. Our approach solely works on raw pixels without requiring any extra tricks. Dabkowski & Gal (2017) also train a separate neural network dedicated to predicting saliency maps. However, their approach is a classifier-dependent method and, as such, a lot of effort is devoted to preventing generating adversarial masks. Furthermore, the authors use a complex training objective with multiple hyperparameters which also has to be tuned carefully. On a final note, their model needs a ground truth class label which limits its use in practice.

CONCLUSIONS
In this paper, we proposed a new framework for classifier-agnostic saliency map extraction which aims at finding a saliency mapping that works for all possible classifiers weighted by their posterior probabilities. We designed a practical algorithm that amounts to simultaneously training a classifier and a saliency mapping using stochastic gradient descent. We qualitatively observed that the proposed approach extracts saliency maps that cover all the relevant pixels in an image and that the masked-out images cannot be easily recovered by inpainting, unlike for classifier-dependent approaches. We further observed that the proposed saliency map extraction procedure outperforms all existing weakly supervised approaches to object localization and can also be used on images containing objects from previously unseen classes, paving a way toward class-agnostic saliency map extraction.

APPENDIX ARCHITECTURE AND TRAINING PROCEDURE
Classifier f and mapping m As mentioned before, we use ResNet-50 (He et al., 2016) as a classifier f in our experiments. We follow the encoder-decoder architecture for constructing a mapping m. The encoder is implemented also as a ResNet-50 so its weights can be shared with the classifier or it can be separate. We experimentally find that sharing is beneficial. The decoder is a deep deconvolutional network that ultimately outputs the mask of an input image. The input to the decoder consists of all hidden layers of the encoder which are directly followed by a downscaling operation. We upsample them to be of the same size and concatenate them into a single feature map H. This upsampling operation is implemented by first applying 1×1 convolution with 64 filters, followed by batch normalization, ReLU non-linearity and then rescaling to 56×56 pixels (using bilinear interpolation). Finally, a single 3×3 convolutional filter followed by sigmoid activation is applied on H and the output is upscaled to a 224×224 pixel-sized mask using proximal interpolation. The overall architecture is shown in Fig. 1.
Training procedure We initialize the classifier f (0) by training it on the entire training set. We find this pretraining strategy facilitates learning, particularly in the early stage. In practice we use the pretrained ResNet-50 from torchvision model zoo. We use vanilla SGD with a small learning rate of 0.001 (with momentum coefficient set to 0.9 and weight-decay coefficient set to 10 −4 ) to continue training the classifier with the mixed classification loss as in Eq. (8). To train the mapping m we use Adam (Kingma & Ba, 2014) with the learning rate l 0 = 0.001 (with weight-decay coefficient set to 10 −4 ) and all the other hyperparameters set to default values. We fix the number of training epochs to 70 (each epoch covers only a random 20% of the training set).

RESIZING
We noticed that the details of the resizing policy preceding the evaluation procedures OM and LE vary between different works. The one thing they have in common is that the resized image is always 224×224 pixels. The two main approaches are the following.
-The image in the original size is resized such that the smaller edge of the resulting image is 224 pixels long. Then, the central 224×224 crop is taken. The original aspect ratio of the objects in the image is preserved. Unfortunately, this method has a flaw -it may be impossible to obtain IOU > 0.5 between predicted localization box and the ground truth box when than a half of the bounding box is not seen by the model.
-The image in the original size is resized directly to 224×224 pixels. The advantage of this method is that the image is not cropped and it is always possible to obtain IOU > 0.5 between predicted localization box and the ground truth box. However, the original aspect ratio is distorted.
The difference in LE scores for different resizing strategies should not be large. For CASM it is 0.6%. In this paper, for CASM, we report results for the first method.

VISUALIZATIONS
In the remained of the appendix we replicate the content of Fig. 2 for sixteen randomly chosen classes. That is, in each figure we visualize saliency maps obtained for seven consecutive images from the validation set. The original images are in the first row. In the following rows masked-in images, masked-out images and inpainted masked-out images are shown. As before, we used two instances of CASM (each using a different thinning strategy -L or L100) and Baseline.


Categorizing and Labelling of Twitter Dataset using RCNN Model

: The fast advancement in the Web, web based business and interpersonal organizations realizes a lot of client created messages on the Web, for example, online surveys for items, administrations and web journals. Such messages as online audits are typically subjective and semantic arranged. The semantic introduction to separate and classify such messages appropriately is of incredible research and commonsense esteem. Order of writings is trying due of limited logical data and the scanty semantic data they regularly contain. The current research on content arrangement of writings fundamentally incorporates include based methods. The advancement of profound learning models has accomplished astounding outcomes in PC vision and discourse acknowledgment. In this content categorization method we will actualize the use of Convolution Neural Systems (CNNs) and Intermittent Neural Systems (RNNs) which acknowledges dynamic length input utilizing online networking information, for example, twitter. The proposed framework expects to build the execution measurements, for example, time and exactness to over 80% contrasted with other profound learning strategies. bag-of- for for seq-CNN for content for of seq-CNN different utilizing three datasets in particular Motion picture Audits (IMIDB) 8.74,

I. INTRODUCTION
Content classification is the undertaking of appointing predefined classes to content records. It gives applied perspectives of reports and is utilized as a part of different applications in reality. For instance, cooking formulas are commonly sorted out by kind of fixings utilized, having a place with different land locales, formulas given by individual gourmet experts, scholastics can be classified to various spaces, therapeutic reports can be arranged to various tests et cetera. Spam separating is another application for content order where we channel or sort messages which are not to our significance. Content classification depends on words in which straightforward measurements of the mixes of words are performed till date. On the other hand, different analysts have discovered profound taking in valuable in isolating data from crude information running from PC vision to discourse acknowledgment to content arrangement. Characteristic dialect handling (NLP) has helped a great deal from the restoration of profound neural systems (DNN). There are different sorts of DNN designs in particular Recursive Neural System (RecursiveNN), Repetitive Neural System (RecurrentNN) and Convolutional Neural System (CNN). To beat the impediments of RNN calculation two new sorts of calculations have been created to be specific Long here and now memory (LSTM) and Gated Intermittent Unit (GRU) [1] [4]. The following are the various models in deep learning:

A. Recursive Neural Network (RecursiveNN):
RecursiveNN has turned out to be effective in building sentences. The RecursiveNN grab the clarification of a sentence utilizing a tree structure. The execution of this model wholely relies upon the portrayal of the tree structure giving the time multifaceted nature of O(n2) where n is the length of the content. Utilizing this model could be tedious at the point when the model takes a shot at long sentences and even relationship betweentexts is difficult to speak to utilizing a tree structure. Along these lines it is expressed that demonstrating long sentences by recursive neural system is deficient [2]. The principle preferred standpoint of recursive neural organize is that it is proficient in building sentences anyway it is wasteful to demonstrate long sentences and reports. The execution of this model relies upon the literary tree structure and along these lines the execution corrupts as the tree structure increments relying on the content.

B. Recurrent Neural Network (Recurrent NN)
Recurrent NN is a model that gives better reasonable data and is consecutive structures. It breaks down a word and stores the data of all the first words in a settled measured shrouded layer. This model is expressed as a one-sided display where all the following words in a sentence are successful than the past one because of which the data is accessible toward the finish of the report. Be that as it may, this can be an issue as the powerful data won't generally be available toward the finish of the report and the model uses its interior memory to process subjective arrangements of information sources. The time many-sided quality of RecurrentNN is O(n). RNN has the catch the logical data and can catch semantics of long messages as it can deal with self-assertive information and yield lengths. The utilization of RNN is for the most part utilized for content and discourse examination [3].

C. Convolutional Neural Network (CNN)
CNN reports higher execution contrasted with RecurrentNN. CNN processes the most educational n-grams of the considerable number of words in a sentence and concentrates the most profitable data. CNN can be utilized of estimation arrangement as the feelings are controlled by key expressions. It is an unprejudiced model and uses sustain forward ANN to reasonably segregate phrases utilizing a maximum pooling layer. In this manner CNN can catch semantic data superior to repetitive and recursive neural system. The time many-sided quality of CNN is likewise O(n). Be that as it may, the past examinations tend that CNN utilize straightforward convolutional bits because of which it is hard to decide the measure of the window. . The utilization of CNN is for the most part utilized for picture and video handling [4].

D. Recurennt Convolutional Neural Network (RCNN)
To defeat the negative marks of the above model RCNN was presented for content arrangement. To start with, the bi-directional intermittent structure is connected which acquaint less commotion contrasted and other customary window based neural system and subsequently it gives better relevant data. Second, we present a maximum pooling layer which decently figures out which include assumes a vital part contrasted with different words in the sentence. Consequently, the RCNN is a blend of repetitive neural system and convolutional neural system with a period multifaceted nature of O(n) [4].

II. RELATED WORK
CNN is joined with LSTM without the technique for word division. The technique utilized was Burn CNN in view of Zhang's display where k-max pooling is actualized rather than max-pooling. Max-pooling is a technique for down testing where a sliding window is utilized on a column and chooses a cell with most extreme esteem and after that the window is passed to next layer. [5] Kmax pooling on the other hand doesn't have a window rather it performs choosing activity for the whole column. The best k esteems with greatest esteem is chosen and go to the following layer. In this way, the creators have utilized Singe CNN technique by utilizing k-max pooling which has the ability to acknowledge any length of contribution before a completely associated layer and gives preferable exactness over other conventional word-level strategies. This technique can be connected to other common dialects utilizing word division for future research. In [2], the creator has connected RCNN an intermittent structure to catch the relevant data utilizing a bi-directional repetitive structure alongside the maximum pooling layer which removes the most significant element and key parts in the content or report. Utilizing this one-sided display the later words are more overwhelming than the past one. In any case, the viability is decreased as the key segments won't generally be available toward the finish of the archive as the basic convolutional bits are hard to decide the window measure. Henceforth, it is imperative to decide the window measure for a basic convolutional portion to decide the highlights of a document. In [3], the creators have given a methodical examination of the profound learning models, for example, CNN, LSTM and GRU utilized as a part of characteristic dialect preparing and expecting to choose the most suitable profound neural system models. The strategies actualized contains the preparation information utilizing fundamental setup without complex traps and have utilized for scanning parameters for each demonstrate independently. The examination specifies that the CNN show utilizes convolutional layer and GRU models the contribution from left to right and utilize the last layer as the after effect of the info. Because of this the variety in the bunch estimate and shrouded measure causes wavering which can be overcome in the event that it can be enhanced to expand the execution of the models. The utilization of CNN display removes lexical and sentence level highlights. The model takes all the word tokens and changes to vectors utilizing word embeddings. The lexical highlights are then removed by the things introduce in the sentence. After the extraction both the techniques are joined to deliver the last component vector. To determine which combine of words should be marks to the separate word. The creator has utilized SemEval-2010 Errand 8 dataset which is uninhibitedly accessible and contains 10,717 examples alongside 8,000 and 2,717 preparing what's more, test occasions individually. [4]. In [5], the creator has utilized different extensive scale dataset, for example, AG's news with 4 classes, 120,000 what's more, 7,600 prepare and test tests individually alongside different datasets and actualized in the convolutional neural system. The examination of these dataset is among the different other conventional models, for example, pack of-words, n-grams and other profound learning models.
There are two kinds of CNN in particular, clear adjustment of CNN from picture to content and second is basic CNN utilizing bagof-words. There different strategies used to arrange utilizing CNN will be CNN for picture, CNN for content, seq-CNN for content and bow-CNN for content. The correlation of mistake rates (%) appeared in the paper, demonstrates that seq-CNN beats different strategies for classification utilizing three datasets in particular Motion picture Audits (IMIDB) with 8.74, Hardware Item Surveys (Elec) with 7.78 and News Articles (RCV1) with 9.96 [6]. Utilizations one-sided demonstrate. Be that as it may, the viability is decreased because of confusion of segments toward the finish of the record. [2] RCNN is utilized as the mix of utilizing the b-directional repetitive system alongside a maximum pooling layer which consequently gives the key parts in the document.
Determine the size bits in CNN.
Variety in group estimate and shrouded measure causes wavering. [3] The strategies executed: Preparing information utilizing fundamental setup without complex traps. CNN utilizes convolutional layer then again GRU gets the contribution from left hand side and gets the outcome from last layer.
Optimization of concealed size and bunch size to expand the execution of the models.
Determination of which name to be allocated to match of words. [4] Explore the connection and concentrate the lexical highlights.
It likewise determines which highlight should be marked.
In the paper SemEval-2010 Undertaking 8 dataset is used.
It physically picks the best list of capabilities and increment the execution.
Less accomplishment in the change of data recovery [5] Using CNN on different dataset and make two Conv-Nets one expansive and other little and introduce the weight utilizing Gaussian distribution.
Apply CNN to a huge scope of dialect preparing undertaking.
1D structure of content information should misuse for precise anticipate work. [6] Two strategies to be specific straight forward CNN and pack of-words change are applied.
Increasing the proficiency of CNN show with the assistance of sack of-n-grams approach.

III. PROBLEM STATEMENT
In RNN the associations among units are as coordinated cycle that makes an inward state which enables the system to take a shot at dynamic conduct. Because of this RNN can characterize intermittent association in light of timestamps among different states. This issue in CNN models are utilized which utilizes convolutional layers called as convolutional layers. CNN can suit for content classification however are less noticeable in taking in the data from the information gave in the content. The CNN show is an unprejudiced model which utilizes the feed forward system and furthermore utilizes max-pooling layer rather than k-max pooling. This issue articulation is to beat every one of the defects said in the paper.

Fig. 1. Proposed System Architecture
In the proposed framework, is partitioned into different stages: 1) Phase 1: The tweets are downloaded from twitter Programming interface which is in .json arrange. The downloaded tweets should be changed over into .csv arrange by pre-handling it for additionally preparing of the dataset. 2) Phase 2: The changed tweets request to be prepared by marking it in particular infected, preventive, informative and others.

3) Phase 3: Once the information is prepared the dataset is given to the models and loss function is figured 4) Phase 4:
The yield of the model is tried relying upon the loss function and the preparation gave with respect to the figure shown below. The train model utilizes the dimension vector for each word and the dataset is split into training data and test data along with dev data. A directory is created which stores the data that is trained by the model along with the timestamp of that training instance. The model is then evaluated using the training data and the test samples i.e., the tweets. For every batch the data is trained and stored in the directory that is categorized as description and the category. Once the data is trained the next step is to predict the models using the test data that are random flu tweets extracted from the Twitter API. The .csv file is loaded in the model along with the predicted and description as the heading. The predictions are then classified dependending on the dictionary that is feed to the system and with the help of trained data the tweets are classified as preventive, infected and information. The contribution to the model is given through preparing set. This information is given to the genuine target name and the neural system. The yield from the objective name and neural system is indicated as y and ŷ individually. Once the yield from both the squares have been gotten we process the forecast is right to what level i.e., ŷ is contrasted and y and it is called as misfortune work. A misfortune work figures the disparity amongst y and ŷ is coordinated to as l as appeared in Fig 4. [9]

V. DATASET
In the proposed strategy we utilize tweets from Twitter Programming interface. The dataset contains tweets of swine influenza or simply expressed as influenza which are refined from the removed tweets. An aggregate of 10,48,576 tweets are gathered similarly for five classes that is tainted, preventive, useful and others. The information is prepared utilizing the data above and the tweets are then named relying on the pre-characterized names gave amid preparing.

VI. RESULTS AND DISCUSSION
The lead of the proposed CNN calculation is contrasted and other existing calculations. The outcome is appeared in Table V. Our anticipated model has a superior exactness than RNN and LSTM models. Be that as it may, the best model in this examination is dynamic CNN (DCNN), which creates more exactness than our model around 5% [1][3] [16].   We have inspected the model utilizing character input. The technique is directed by utilizing the mix of max and k-max pooling as said in the model depicted previously. The outcome in LSTM is awful contrasted with CNN as the criticism is likewise added to the model utilizing an overlook entryway which avoids the weights on the off chance that it is immaterial. CNN despite what might be expected is a fair model uses max-pooling and catches preferable semantic information over different models.

VII. CONCLUSIONS
The outlined model is developed for proficient and compelling grouping to acknowledge variable length input. To perform classification errand in light of twitter information, the dataset is prepared utilizing Convolutions calculation. The CNN demonstrate contains input layer, gathering of units in a system named as widths, accumulation of widths (concealed layers) and a last layer. Our planned model which can acknowledge a more drawn out information gives precision in more superb way than a unique model with a settled length input. It likewise beats other neural system techniques e.g. LSTM, RNN aside from a word-level CNN. In future, CNN can be created to improve and adequately prepare the consecutive information for better execution.