ALMARVI Execution Platform: Heterogeneous Video Processing SoC Platform on FPGA

The proliferation of processing hardware alternatives allows developers to use various customized computing platforms to run their applications in an optimal way. However, porting application code on custom hardware requires a lot of development and porting effort. This paper describes a heterogeneous computational platform (the ALMARVI execution platform) comprising of multiple communicating processors that allow easy programmability through an interface to OpenCL. The ALMARVI platform uses processing elements based on both VLIW and Transport Triggered Architectures (ρ-VEX and TCE cores, respectively). It can be implemented on Zynq devices such as the ZedBoard, and supports OpenCL by means of the pocl (Portable OpenCL) project and our ALMAIF interface specification. This allows developers to execute kernels transparently on either processing elements, thereby allowing to optimize execution time with minimal design and development effort.

Introduction
Image and video processing is an important enabler for a large number of application domains ranging from medical imaging to entertainment. These applications continue to push the requirements imposed on our computational Tampere University of Technology, Tampere, Finland 3 UTIA, Prague, Czechia platforms to their technological limits, both by increasing the demand for higher resolution images to be processed on the one hand, and by increasing the computational complexity of newly proposed processing algorithms on the other. As a result, applications constantly push beyond the computational capacity that general purpose computational systems can provide, and demand the design of high performance processing hardware that is customized towards the targeted video application.
Various different types of image and video processing platforms are introduced regularly to match the ever increasing requirements of the application domain. Each platform is optimized to specific types of applications, and focuses on a given set of optimization criteria. However, creating a customized processing platform is a long and costly process, both in terms of hardware design time as well as application porting time to these platforms. Some platforms provide custom made accelerators to ensure high performance, but suffer from long and expensive hardware design time [1]. Other platforms provide optimized processing cores for easy programmability, but only ensure performance for the specific targeted application [2].
In this paper, we present the ALMARVI execution platform which enables both hardware design flexibility as well as portable software deployment. The hardware architecture allows for integrating heterogeneous processing cores each optimized for their own application, thereby ensuring optimized performance. At the same time, the software stack allows for portable deployment of applications on each core based on an OpenCL programming layer [3] that has broad support in the industry for a whole range of computing platforms: multicore CPUs and GPUs, in addition to the upcoming standard of automatic hardware synthesis right from OpenCL code.
In order to facilitate integration of components (such as newly developed hardware and licensed processors and IP blocks) and to connect to cross-compatible OpenCL compilers, a new common hardware integration interface (called AlmaIF) was developed. Additionally, we present details of the implementation and integration of two heterogeneous acceleration fabrics and show how the same program implemented and compiled using a single code base can be executed on a variety of execution platforms. Software compatibility is provided by adding support for the AlmaIF interface to the Portable OpenCL environment.
This paper is organized as follows. Section 2 discusses the background and related work. Section 3 shows how the AlmaIF abstraction layer is able to virtualize away the details of the custom processing cores used in this paper. The design flows of both processors used in the prototype are discussed in Section 4, along with a detailed example on how to implement the AlmaIF interface on a processor. The hardware implementation and software support for running OpenCL programs on the platform are discussed in Section 5. We report preliminary numbers of a case study application running on the ALMARVI execution platform (an OpenCL implementation of the Sobel filter) in Section 6. Finally, Section 7 ends with the conclusions.

Background
Two types of processing units are used in this paper to showcase heterogeneous integration capabilities of the ALMARVI execution platform. The first supported custom processing units that supported by the platform are based on the Transport Triggered Architectures (TTA) paradigm [4]. They are generated by the TTA-based Co-design Environment (TCE), an open-source project maintained by Tampere University of Technology (TUT) [5]. These represent cores that are not just customizable regarding certain design parameters; the entire architecture of the datapath can be designed from the ground up to target an application or application domain. This includes application-specific functional units but also involves generic units such as register files, Arithmetic Logic Units (ALU), etc. These units can be configured individually (for example, register files can have different numbers of access ports -a highly influential design parameter in processor design [6]). The connectivity between each of the functional units is also a design decision. TCE generates a compiler and simulator for the processor and all of these designtime configurations can be performed using a graphical user interface.
The second processor type is the ρ-VEX reconfigurable VLIW processor, maintained by Delft University of Technology [7,8]. The core has a range of design parameters that can be changed at design-time to target certain application domains. In addition, it can be reconfigured during run-time to adapt itself to the workload, making it a highly adaptive general purpose core [9]. This is realized by 1) designing a flexible connection between the processor's datapaths and its program state storage, and subsequently 2) multiplying the state storage so that multiple programs can execute simultaneously (similar to Simultaneous Multithreading -SMT [10]). Datapaths can be assigned to program states, effectively splitting and merging the processor into one or multiple cores. They can execute a single program at high performance or multiple programs at high total throughput. The decision to split can be based on whether an application has large amounts of Instruction-Level Parallelism (ILP) or Thread-Level Parallelism (TLP).
In order to integrate these two processing units together in the same platfrom, we provide the ALMARVI Common Hardware IP Interface (AlmaIF) to enable plug'n play style of customization of the hardware platform at the system level, by allowing adding accelerators to a new hardware platform design with an easy integration to the OpenCL-based system software layer. The AlmaIF interface abstracts away implementation details so that application development only needs to focus on the OpenCL implementation. This article gives an overview of the ALMARVI execution platform used to demonstrate the capabilities of AlmaIF, which can be used for easily integrating any hardware component that supports the interface.

Related Work
A large body of work exists that target FPGA not for prototyping purposes, but for acceleration. Prior efforts that implemented FPGA acceleration fabrics include [11], a project by Microsoft that added FPGAs to a datacenter running the Bing service. These FPGAs were equipped with programmable accelerators (softcore processors) that are highly optimized for the specific application domain. Other projects using programmable accelerators include [12] for a FFT workload and [13,14] in the image processing application domain. These related efforts do not use OpenCL and/or heterogeneous accelerators. In [15], an exploration is performed regarding how much acceleration can be achieved using accelerators while still retaining programmability. A Heterogeneous accelerator fabric for accelerating OpenCV is presented in [16].
Within the area of FPGA accelerators, a current topic is to support high-level languages such as the one we are also using in this work, OpenCL. For example, in [17], Xilinx presents an OpenCL library of streaming acceleration units, allowing designers to quickly generate an FPGA acceleration fabric with OpenCL support. Additionally, Xilinx has introduced SdAccel, a toolset to generate hardware from OpenCL kernels [18]. An Altera equivalent is presented in [19]. In [20], a rapid prototyping framework for video processing using High-Level Synthesis is presented. RIFFA [21] is an opensource integration framework that provides an interface to connect to FPGA accelerators via PCIe. A related industry-effort aimed towards improving integration of various execution platforms is the Heterogeneous System Architecture (HSA) [22].

AlmaIF Interface
The targeted system architecture is shown in Fig. 1. The system has a single AXI interconnect, where each accelerator is connected as a slave device. A closer view of one accelerator is shown in Fig. 2a. Accelerators may also be multi-core as shown in Fig. 2b. Each core has a separate debug and control interface.
The master can access four memory sections in each accelerator: -Control interface (CTRL). Registers used to query basic information about the accelerator and its status, and to start and stop execution. Each core in an accelerator has a separate control interface.

Figure 2
Accelerator architecture for a single core and a multicore system. These are mapped to the accelerator slave's address space such that a memory address has a field long enough to store a byte address into the largest section; followed by two high bits to select between sections as shown in Table 1. The size of the AXI IP memory address can be computed as: The sizes can be queried through the control interface. For example, the reference TTA design has the following memory resources: -32KB instruction memory (4096 instructions × 8 bytes per instruction) = 15-bit address -32KB data memory = 15-bit address -2KB parameter memory = 11-bit address -Single core with a minimum-size Resulting in a total address size: 2 + max(8,15,15,11) = 17, and the memory map shown in Table 2. Accesses to memory locations between sections (e.g. 0x00400) have undefined behavior. The design rationale here is that given the base address of the AXI slave, a driver program can probe memory section sizes from CTRL (which is the first memory section), and use accelerators with differently sized memories in a plug-and-play manner.
The device identification part of the CTRL region allows the software layer to identify what kind of accelerator it is accessing. This is needed to determine which compiler backend is needed to compile kernels for the accelerator, or, in the case of offline compilation, which binary to use. Accelerators are identified by a device class and a device ID. The device class specifies the kind of accelerator (for Table 2 Example memory address regions for a single-core TTA platform. 0x00000 .. 0x003ff CTRL 0x08000 .. 0x0ffff IMEM 0x10000 .. 0x17fff DMEM 0x18000 .. 0x187ff PMEM instance, a TTA, a ρ-VEX, etc.), while the device ID should uniquely identify its configuration (for instance, in the case of a ρ-VEX, the issue width). The capability registers contain information about the size of the memories, the number of cores within the accelerator (each core has its own copy of the CTRL region), the supported debug features (single-stepping and/or hardware breakpoints), and an AlmaIF version number. The latter allows AlmaIF to be extended later, with a backwards-compatible software layer.
The status registers supply the software layers with runtime information, such as whether the accelerator is running or not, the current program counter, and performance information. The command registers allow the software layers to reset the accelerator, to start or stop execution, to set the start address, and to set breakpoints for debugging purposes.
The instruction memory holds the kernel binary that the accelerator should execute. The data memory is intended to be used for variables local to the accelerator, which may be initialized by the host, and/or exchange of data between the accelerator and the host. The parameter memory is intended to be used as a command queue for the kernel running on the accelerator.
The CTRL memory region provides an interface according to the AlmaIF specification. Support for this interface has been implemented in the pocl runtime system, so that any accelerator that conforms to the interface can be used to run OpenCL kernels via pocl, as will be discussed in following sections.
One of the key points for the AlmaIF is to provide a hardware layer that is generic enough for sharing driver code in the OpenCL implementation layer. A hardware abstraction such as AlmaIF makes it easier to plug in different AlmaIF-compliant devices in a compute platform with only minimal changes to the driver code. This helps in making the ALMARVI execution platform composable: as devices share a common control interface, it reduces the integration effort at multiple levels due to not requiring device-specific knowledge at the simplest control functions for verification and execution purposes, for instance. The pocl runtime that utilizes AlmaIF as a hardware abstraction layer to control the execution is described in more detail in [23].

Fully Customizable Cores: TCE
Processor customization in TCE is usually conducted as a hardware-software co-design process. The processor design is iterated by varying the processor template parameters defined using the TCE files while adapting the software to better exploit the features, such as by calling custom operation intrinsics or enhancing parallelization opportunities. An example processor design is depicted in Fig. 3. The co-design process is supported by a set of tools, as illustrated in Fig. 4. Initially, the designer has a set of requirements and goals placed to the end result. It is usual to have a real time requirement as the primary requirement, given as a maximum run time for the programs of interest and the secondary goal is to optimize the processor for low power consumption or minimal chip area. In some cases, there can be a strict area and/or power budget which cannot be exceeded, and the goal is to design as fast processor as possible that still fits below the limit. The iterative customization process starts with an initial predesigned architecture. Using a graphical user interface tool called Processor Designer (ProDe), the designer can add, modify, and remove processor components. The connectivity between the components can be customized manually using the GUI or with an automated optimizer. Each iteration of the processor is evaluated by compiling the software of interest to the architecture using TCE's re-targetable high-level language compiler and simulating the resulting parallel assembly code using an architecture simulator. The simulator statistics give the runtime of the program and the utilization of the different datapath components, indicating bottlenecks in the design. The architecture exploration cycle enables low effort evaluation of different custom operations. For a completely new processor operation, the designer describes the operation simulation behavior in C/C++, estimates its latency in instruction cycles when implemented in hardware, and adds the operation to one of the function units in the architecture. This way it is possible to see the effects of the custom hardware to the cycle count, before deciding whether to include it in the design or not. When a design point fulfilling the requirements has been found, or more accurate statistics of a design point is needed, the designer uses a tool called Processor Generator (ProGe), which produces an RTL implementation of the processor.

Automatic Generation of TTA Accelerators
The main components of TCE that provide substantial automation to the design flow are: -Processor Generator (ProGe), which produces an RTL implementation of the designed processor, and -Design Space Explorer (explore), which can be used to perform automated transformations on TTAs.
RTL Generation-Thanks to the modular TTA template, the RTL generation is straightforward and reliable; the generator utility ProGe collects component implementations from a set of hardware databases (HDBs) and generates the interconnection network which connects the units. Instruction fetch and decoding logic is generated based on the given processor architecture description file. The designer has to implement any new function units in VHDL or Verilog and add them to an HDB. A tool is available to automatically validate the function unit implementation against its architecture simulation model. The generated RTL is fed to a standard third party synthesis and simulation flow. This step generates more detailed statistics of the processor at hand, which can again drive the architectural exploration process. The detailed implementation level statistics map trivially back to the design actions at the architecture level. For example, required chip area can be reduced by removing architecture components. Similarly, adding more pipeline stages to complex function units, or reducing the connectivity of the interconnection network helps increasing the maximum clock frequency.
Design Space Exploration-TCE includes a plugin-based framework for automatic design space exploration. The framework is built around a design space database (DSDB) which stores processor configurations: plugins can then be invoked on a configuration to generate one or more modified configurations. Plugins are guided by the simulated performance of the design in benchmark programs, and by an area and power model characterized on a 130 nm ASIC technology. Two examples of plugins are GrowMachine which adds computational resources to the TTA until no more parallelism can be extracted from the benchmark programs, and ShrinkMachine which removes as many resources as possible while honouring a real-time constraint. The framework includes sufficient functionality to generate processors from scratch, but in practice a better quality is currently obtained by hand-crafting the design, while using the explorer to automate repetitive tasks. Explore has plugins designed to produce efficient TTA interconnection networks (ICs) [24]. The IC is particularly tedious to design by hand, and has a very large design space which poses difficulties for automated search. The main idea is to start from an IC topology identical to a conventional VLIW, and then shrink it through repeated bus merge and register file port merge operations. The resources to be merged are chosen by profiling the benchmark workload and generating a resource utilization covariance matrix: resources with low covariance are seldom used simultaneously and, therefore, can be merged without impacting the performance much. The resulting automated process is shown in Fig. 5.

Configuration of TTA Accelerators
A library of components has been built for integrating TTAs with AXI interfaces, including caches and Load/Store Units which communicate over AXI. Tasks such as starting and stopping execution are done through a debug and control interface, usually accessed through an AXI slave peripheral. The interface can currently be used to clock gate an unused accelerator by placing it in a break state; it would also be straightforward to add further low-power modes. The interface also supports debug features such as multiple breakpoints, stepped execution, and examination of various performance counters and internal state data of the processor. In order to facilitate testing of ASIC prototypes with a limited number of IO pins, a JTAG interface wrapper gives access to memories and control interfaces over a fourwire interface. See Fig. 6 for the default organization of a rvex pipeline.

Adaptable cores: ρ-VEX
The ρ-VEX is a dynamically reconfigurable VLIW processor. It is implemented in VHDL and can be synthesized to FPGA targets for prototyping. The reconfigurable concepts used in the core can also be used in ASIC technology as they do not require the reconfigurable properties of the FPGA. The VEX instruction set used in the processor is created by HP. Its target application domain is media applications (image/video). The core is designtime reconfigurable through VHDL generics in a number of properties: Issue width, functional unit layout, cache sizes, etc. Run-time reconfigurability can also be disabled to save chip area. The run-time reconfigurable core can change its issue width dynamically by splitting the core into smaller subcores that have their own execution context. This way, the processor aims to run a wide range of workload types in the most efficient way possible. Single threaded applications with a high level of parallelism in the code can run on a large high-performance core, and multi-threaded applications can run in parallel on smaller cores. The core supports a number of execution contexts that can be assigned to pipeline pairs (lanepairs). Both properties are design-time reconfigurable. When a single context is assigned to all lanes, the core runs in the widest configuration. By assigning different contexts to the lanepairs, they will each run a separate program (see Fig. 7). The ρ-VEX uses memory-mapped registers to control its configuration. Supervisor software can load a set of programs into the execution contexts, and use the configuration control registers to select which programs are being executed and in which configuration. The contexts contain all information needed to resume execution when being suspended (for example, when all the lanes are assigned to another context). This includes all registers, program counter, trap/interrupt state etc.
The ρ-VEX platform consists of the synthesizable VHDL designs of the core, cache and a number of peripherals. On the software side, there is an interface tool that can connect to the core and provides the user with extensive debugging capabilities and full control of the processor. The implementation of support for AlmaIF is discussed in Section 4.2.2. There are a number of different compilers that can target the ρ-VEX, there is a port of binutils and GDB, there is basic Linux support (uCLinux with a NOMMU kernel), runtime libraries (the uCLibc C standard library and newlib) and an architectural simulator.

Design-Time Customization
The ρ-VEX processor is both run-time and design-time configurable. When creating a general-purpose platform Figure 7 ρ-VEX datapaths and program state storage areas (contexts). In the depicted configuration, two pairs of datapaths are assigned to context 2, consequently it is running in 4-issue VLIW mode. The other two pairs are assigned to task 1 and 4, who are both running in 2-issue mode. Context 3 is currently disabled. Switching between configurations can be done at the cost of a pipeline flush. that must be able to provide high performance for single threads and high throughput for multiple threads, the full dynamic 8-issue core can be used. However, the area utilization of this configuration is high. If the workload is highly parallelizable, a static 2 or 4-issue core configuration can be used. The area utilization is considerably lower, and single-thread performance can be sacrificed because the workload will rely on multithreading to achieve high performance. The reduction in area allows more core to be placed on the chip, resulting in higher total throughput. In addition to design-time configuration of the processors, multiple ρ-VEX cores can be connected to each other using local scratchpad memories, preventing the need to access global memory.

AlmaIF Integration with pocl on ρ-VEX
As an example of how the AlmaIF interface may be implemented in hardware for an existing accelerator, we report the implementation for the ρ-VEX processor in more detail.
The ρ-VEX processor is a VLIW processor, with configurable amount of virtual processors, known as contexts. A control unit known as the reconfiguration controller manages the division of the available computational resources amongst these contexts. All contexts share a common instruction memory, and a common data memory. To allow the processor to be controlled externally, for debugging or otherwise, a number of control registers are available through a debug port.
This was mapped to AlmaIF as follows. First, an additional access port to the instruction and data memories was added to allow the host processor to access these through the respective AlmaIF memory regions. As the ρ-VEX has only one data address space, the AlmaIF parameter memory region was simply mapped to the ρ-VEX data memory as well, making the size of the ρ-VEX data memory the sum of the AlmaIF data and parameter memory sizes.
The ρ-VEX debug port serves a similar purpose as the AlmaIF control region. However, the ρ-VEX and AlmaIF registers do not map one-to-one. Therefore, a translation unit was constructed that emulates the AlmaIF registers. In order to still allow the ρ-VEX debug port to be used natively, it was also mapped to AlmaIF directly, using a memory region not specified by AlmaIF.
Each ρ-VEX context has its own set of debug registers. Therefore, each ρ-VEX context is represented as an AlmaIF core.

Hardware
To generate a SoC platform, IPs need to be generated from the accelerator's sources that can be used by Vivado to integrate into a AXI-based project. These steps can be performed according to the accelerator's documentation. Note that for the accelerators that are available at the time of writing, certain levels of configuration are available when generating the IPs. The TTAs that are generated from the TCE (TTA Co-Design Environment) are fully customizable. A graphical user interface is available to accomplish this, see (http://tce.cs.tut.fi/user manual/TCE/) for more information. The ρ-VEX has a number of designtime options that are available before generating the IP (by means of a configuration script), and additional options that are implemented as VHDL generics that can be set in Vivado after importing it. The options that are handled by the configuration script are for example the pipeline organization and control registers, and the VHDL generics include issue width, memory/cache sizes and performance counters.
After adding the desired IP blocks to a Vivado project, the user needs to assign address spaces according to the memory regions discussed in Section 3 and customize the clocking. This should be done considering the configuration of the accelerators, as this influences the achievable timing considerably. For instance, a 2-issue ρ-VEX with 7stage pipeline and forwarding disabled can be expected to operate on much higher frequencies compared to an 8-issue configuration with forwarding and a shorter pipeline.

Software
The AlmaIF interface was added as a target in the pocl runtime system in order to support the execution of OpenCL programs. This section discusses the way in which this can be enabled during compilation of the pocl runtime itself and how programmers can offload tasks to AlmaIF accelerators.

Compile-Time
The target accelerators need a compiler capable of accepting OpenCL input to generate a suitable binary. Furthermore, an OpenCL command queue mechanism must be provided to have the accelerators check for new work from the OpenCL runtime. The binary contains a field to identify the accelerator it was compiled for, so that the runtime can verify it is loading a proper binary. This field can be used to include the configuration of the accelerator (e.g., the VLIW issue width or pipeline latencies) or a hash which is more appropriate in case of the TTAs that have an enormous configuration space.
The pocl OpenCL framework was extended with an AlmaIF target that is able to drive accelerators. After compiling pocl with the interface enabled, OpenCL programs can be linked together with the pocl runtime library.

Run-Time
At run-time, the user needs to pass the location of the accelerators in the address space. A dedicated Linux device driver that supports bus scanning or other means of hardware detection is future work. The memory addresses can be passed to pocl by means of environment variables. When calling the program, the user needs to supply the appropriate binaries for the accelerators to allow the runtime environment to load them into the accelerator's memories.

Experiments/Evaluation
We use a standardized FPGA-based platform for development and demonstration purposes. This platform consists of a Zynq ZC7020 based FPGA development board with an high-quality camera connected, HDMI interfaces and an FPGA configuration with everything necessary to prototype real time video/image processing solutions at system level.

Target platform: Xilinx Zynq
The Xilinx Zynq series of system-on-chip FPGAs consist of a hard Processing System (PS) with a dualcore Cortex A9 ARM processor, and a varying amount of Programmable Logic (PL). The main resources in the PS are: The HP and ACP ports give better throughput than GP, so they should be used for large data transfers. GP ports should be used for programming accelerators and control traffic.
Using the ACP may simplify programming of the ARM, but it can be inefficient for very large data transfers that thrash the cache and this can significantly slow down the performance of the ARM. The PS is typically clocked at ca. 667 MHz and the practical maximum of PL is in the range of 100-150 MHz. Any IO between the PS and PL has some latency overhead due to the clockdomain crossing (CDC).
We have expanded the prototype FPGA target board, a Trenz carrier board TE0701-5 with Zynq System on Module (SoM) TE0720-2-2if, with a Python 1300 camera module and Avnet FMC IMAGEON HDMI I/O extension board. The complete hardware setup can be seen in Figs. 8 and 9. The FPGA part of the platform contains a video chain running from Python camera input with 1280x1024p60 in 16-bit YUV422 pixel format through a video DMA unit that stores video frames to DDR memory and reads them back independently for video output by a HDMI output interface. The FPGA design of the platform is provided in the form of a Xilinx Vivado project and supports custom hardware accelerator extensions. Software support for the platform is built on Petalinux. The platform contains drivers for its hardware components located in FPGA fabric and a simple sobel filter example (implemented as C code for ARM) demonstrating how to initialize video stream using drivers and how to use video DMA to process individual frames and synchronize input and output video streams.
Detailed descriptions and an evaluation version of the platform is provided by UTIA as a downloadable package (http://sp.utia.cz/index.php?ids=results&id=apcp).

Multi-accelerator Prototype
For the integration prototype, an FPGA design is created with two TCE cores and one 4-issue ρ-VEX core, and   another with two ρ-VEX cores. The TCE cores were clocked at 200 MHz, while the ρ-VEX cores were clocked at 66 MHz. Any combination of two cores can be chosen from the application software by using the standard OpenCL APIs. A Proof-of-concept multi-kernel application software was written for this platform design instance, based largely on the Sobel example. An example TCE processor design was used with two 2-cycle ALUs and a single 3-cycle 32x32b multiplier unit. The configuration can be seen in Fig. 10. The instruction, data and parameter memories have 64 kB, 32 kB and 2 kB of memory, respectively. In addition, the core has 4 kB of scratchpad memory. The organization of the processor is depicted in Fig. 10. The default ρ-VEX configuration used in the ALMAIF platform is a 4-issue reconfigurable core that can split into a 2 core (2-issue). It has 2 hardware contexts (virtual processors) that can hold a full program state. This way, the 2 cores can each run a process independently. The core supports variable-length instruction words to improve the encoding efficiency of the binary code, which results in smaller binaries. The data path consists of 4 ALUs, 4 16x32 multiplication units, 2 MEM (Load/Store) and 2 branch units, depicted in Fig. 11. As discussed earlier, the default size is 32 kB for instruction, data, and parameter memory. The memories have 2 access ports. This is needed in order to provide 1 port per context when running in split 2-issue mode, but it also supports multiple Load/Store operations when running in merged 4-issue mode. The core has full debug functionality including 4 hardware breakpoints and a set of 32-bit performance counters per context. When using ρ-VEX as accelerator, as is the most common use case in this platform, these options may be disabled after debugging and optimizing the application as this will save considerable resources and improve timing. FPGA utilization for the 3-core design is shown in Table 3. The last row shows the total utilization of the FPGA chip's resources.
The example application applies a Sobel filter and a 3x3 box blur filter, in that order, to the image acquired from the camera, and pushes the result to the HDMI output. The filtering is executed on two AlmaIF devices, each handling one filter at the time. The host application handles partitioning the work into smaller segments and enqueuing kernel execution and the required data transfer operations in each device's command queue. The queues are then executed by the pocl runtime, with data dependencies between the two queues resolved by event waitlists. An example of intermediate results and the final result can be seen in Fig. 12.
Although the prototype was made only for proving that the integrated hardware and software stack works, evaluation numbers were produced as a base line for further optimization. For a 1280-by-1024 image, the runtimes for the AlmaIF-based ALMARVI platform instance with different kernels assigned to different device types are depicted in Table 4.

Conclusions
The increasing demand for higher performance at lower energy budgets, combined with the decreasing improvements from technology scaling, leads to increasingly heterogeneous systems where tasks are executed on customdesigned accelerators.
In this context, we implemented an OpenCL platform capable of utilizing multiple different accelerators together in a single program. The pocl OpenCL framework is extended with a target that can drive these accelerators. Because of the generic accelerator interface, the same pocl runtime can be used for any current and future accelerator that adheres to the specification. At run-time, the user can choose which accelerators to enable and the distribution of the kernels to the different accelerators. In the future, this distribution can also be performed by the system based on various metrics such as task characteristics or the system's current power policy.
To the programmer, this fully heterogeneous platform is fully transparent, by making use of the OpenCL framework and compiling for the different accelerators through different compiler back-ends.


extfieldsofstudy:
Computer Science
Physics

text:
Methods of spatial and temporal processing of images in optoelectronic control systems

The data about the effect of image quality on accuracy of information parameters are presented. Studies on accuracy of measuring characteristics of objects in their images depending on image quality are described. Studies showed that accuracy of measurement algorithms improves with an increase in sample sizes of frames in the optoelectronic system (OES). Image quality is improved by processing the frame samples with algorithms for increasing resolution and reducing noise levels, both in single-point and multi-position OESs based on digital cameras with matrix photodetectors. The maximum effect of an increase in information content of the image and accuracy of measuring algorithms is determined by variations in the pixel structure of each image in the region of brightness gradient boundaries. In the single-point OESs which ensure high quality of optical images without noise factors and variations of other external parameters, the optimal number of frames in the OES is four.

Introduction
Images obtained in optoelectronic systems (OES) are used to determine informational characteristics of objects. Currently, various methods are used to improve image quality, including joint processing of images obtained in time sequence with one OES or several independent OESs. Principles of constructing a single-chamber light field (LF) OES are known [1,2]. They record coordinates and direction of the rays. These cameras are equivalent to a large array of low-resolution digital cameras. LF file processing algorithms ensure images with different sets of rays which correspond to a change in the viewing angle of the OES and the depth of the sharply displayed space.
There is a practical issue of improving quality of images to obtain accurate information about the object. Currently, there are approaches to quantitative assessment of quality and complexity of images based on the information theory and the concept of entropy [3]. The complexity of discrete images can be estimated by comparing their two-dimensional variations. A common metric characteristic of variability and complexity of a one-dimensional function [a, b] is a complete variation. A lot of its generalizations to the functions of many variables were suggested (Vitali, Arzela, Frechet, Tonelli's, variations). All of them are variants of the function gradient modulus integral in region [a, b]. In fact, their values are close to real functions. Each variation results in a single value based on the value similar to the function gradient modulus at a point. Based on the generalizations of conclusions and theorems formulated using the above variations, A.S. Kronrod concluded that the function of several r=(x1,x2,…) should be characterized by several independent functionals that determine the length of the boundaries of the components and the number that characterizes severity of local extrema f(r). Complex of the image can be determined by its fractality [4]. The Minkowski's dimension is a way to specify the fractal dimension of a limited set in the metric space. It is closely related to the Hausdorff's dimension [4]. In practice, there may be other methods for assessing complexity and information content of images (e.g., segmentation) [5]. Using the watershed method, the absolute value of the image gradient is considered as topographic surfaces.
A relevant issue is to increase accuracy and reliability of the data about the objects of the OES obtained as a result of the analysis of the pixel structure.
In some cases, image processing methods are used [6] to improve quality parameters of the object's structure and prepare it for the uniform application of measurement algorithms. Measuring algorithms are constructed by analyzing the pixel structure of the image, in particular, curves of the gradient of their brightness [7]. The greatest accuracy in measuring geometric parameters is ensured by integrated methods used for analyzing the contrast gradient, for example, using the wavelet basis [8,9] which reduces the requirements for the pixel structure and the gradient of brightness (contrast) of the edges.
Obviously, using the means for obtaining spatio-temporal (R 3 -t) information from a single-point or multi-position OES and choosing an image processing method, it is necessary to determine possibilities of obtaining the measurement information, its accuracy and reliability, and formulate requirements for the OES.
Due to a large number of configurations of existing OESs, there are no specific recommendations on the optimal choice of means for obtaining and processing information. There is no general method for constructing algorithms for processing and obtaining information parameters of various objects. We studied the effect of image quality on the accuracy of information parameters of the objects. The issues of the need and ways to improve quality of the source images, including in the time sequence, were analyzed.

Materials and methods
We used an OES based on a smart camera by National Instruments (NI) -NI 1742 (CCD 640x480 matrix).
A light field recorder based on a digital camera Lytro ILLUM (version B5-0036 ILLUM) [2] with a CMOS sensor (Aptina MT9F002 14.4 MP, 1 / 2.3 ″, effective image area: 6.14 × 4.6 mm, pixel size 1.4 μm) was used. The number of microlenses in the array is 130,000 (the focal length is 25 microns, the pitch is 13.89 microns). The camera had a main lens of 9.5-77.8 mm with a relative aperture of 1:2. To process light field files, the Lytro Desktop application was used.
To construct the processing and measurement algorithms, we used an application development environment based on virtual instruments NI LabVIEW [10]; an additional NI Advanced Signal Proceedings Toolset module which includes signal processing functions based on the combined frequency-time and wavelet analysis was used. The driver functions of the NI IMAQ Vision technical vision module were used [11]. To capture image frames in real time, the NI Vision Builder for Automated Inspection (Vision Builder AI) application was used, and the NI Vision Assistant application was used to build prototypes of the processing and measurement algorithm.

Studies of accuracy of identification of information characteristics in OES images
The OES with a set of photodiodes (linear, matrix) can ensure a four-dimensional information structure which can be analyzed using the known methods: . As a result of the study of accuracy and reliability of information parameters of the object, the size of the contour was measured using standard algorithms (NI IMAQ-Vision) for finding edges of the brightness gradient. At the first stage, we used algorithms for determining the edge lines yi=tg( with true values taking into account the calibration of the optical system (determination of the linear increase in the optical system). For more accurate measurements of linear dimensions, an algorithm based on the continuous wavelet transform (CWT) of the illumination distribution in the image profile lines [7] was used. It eliminates the effect of the pixel structure of the edges [13]. The results of finding straight edges along the contour edges for a satellite image using the Find Straight Edge algorithm are presented in Fig. 1 and Table 1. An estimate of fractality of the image ( fig. 1a) is shown in Fig. 1b.  In order to determine the effect of image quality improvement on the measuring accuracy, we used the method of combining (adding) a series of images implemented in RegiStax, AutoStakkert, program PIPP and PhotoAcute Studio. Application of the algorithms of these programs increases the resolution of the image, eliminates the effect of noise and random artifacts, temporary mechanical instability of the OES, the gradient of the refractive index of the medium. Moreover, image quality was compared with its fractality and entropy and measuring accuracy of the geometric parameters of the objects. In some cases, measuring accuracy increases as a result of processing a single image, for example, by increasing the local contrast [14]. However, the effect of quality improvement is reduced.
Edges are determined by the re-pixelation of the image structure which depends on the angle of its inclination in the coordinate system of the registrar matrix [13,15]. The nature of the change in the pixel structure is determined by the recorder and sensor's topology. Therefore, accuracy can be improved by scanning the profile line along the inclination angle to the edge [7,13].
The angle of the satellite R as a complex object and its coordinate position (X, Y) were measured by the image of quality KI( ImNxM). Image quality was obtained by decreasing and increasing the resolution (NxM) obtained from a series of images with variable parameters of the pixel structure.
Images obtained by the light field recorder [2,6] were studied. The light field recorder can be considered as an array of low resolution DC [1]. The depth of field in a flat image was ensured by installing a virtual lens aperture. The conversion of volumetric characteristics of an object was specified in the Lytro Desktop application. The dimensional spatial calibration was taken into account [12]. An increase in the resolution of the resulting image in the digital cameras LF is achieved by an algorithm based on the four-dimensional Fourier transformation Ḟ 4 of coordinates (x, y) and the direction of ray propagation [16,1]. Similarly, this algorithm can be used in a series of images of small-sized objects Ḟ 4 {ImK (R 3 -t)} obtained in a sufficient number of images of one OES at various time points or by scanning the image using an OES sensor. The effect of this conversion can increase the resolution of small-sized objects (100 pixels) by more than 100 times. Variations in pixels were determined by their number under microlenses. The use of an image generation system on a single sensor is equivalent to the DC LF principle with an array of microlenses. Processing of the time sequence of images requires solving the problem of their spatial synchronization and changing a significant parameter of variation in the image. Variations in microlens images are determined by directions of the ray [1,16]. Thus, in the registration systems, it is necessary to have or create conditions of variation, for example, a shift in the pixel range, a change in the time of accumulation of illumination (in case of brightness).
The experiments showed that the measurements obtained by summing a series of images from a single-point OES improve measuring accuracy up to 30 % (two images) and 70 % in a multi-position OES. This is due to the pixel blending of the second image. For multi-position OESs, the maximum effect is achieved in two images, after which the errors of transformations and combinations have a strong influence. A further increase in resolution and accuracy occurs when there are random uncontrolled factors and significant noise. Image quality is determined by its fractality which is determined by the number (length) and distribution of the brightness gradient curves. Increased resolution makes it possible to resolve and analyze closely spaced brightness gradient lines ( fig. 2), determine the parameters of smaller details and image fragments. The function «Find edges» by the radial lines (Find Circular Edge) has algorithm settings (Fig 3). It determines coordinates of the contour edges, the circle center for the found polar coordinate system, and the statistical variation of the radius. Varying the parameters of the function allows us to achieve stable results for the shape of the object according to its shape and evaluate its reliability. The number of radial search lines determines the details of the analyzed contour. We can determine spatial coordinates and the size of the satellite by deviation of its shape from the sphere. For images of various quality, the parameters change. Studies have shown the dependence of the measured parameters on the resolution of the original image (Table 2).

Conclusion
After the research of measurement algorithms on images of various quality, we can conclude that increasing resolution of a multi-frame image can improve measurement accuracy by 2-5 times. For small objects with a high noise level, measurement accuracy can be doubled. For the image of the light field obtained by digital cameras in one exposure, accuracy of the measurement algorithms is ensured by the physical size of the sensor. Multi-frame image processing reduces the noise caused by high light sensitivity. Improving quality of the multi-frame processing has an effect when changing the view angle of the camera. Multiposition systems provide more opportunities for obtaining higher quality images, but require knowledge of exact configuration parameters of the OES.


Cross-domain Deep Feature Combination for Bird Species Classification with Audio-visual Data

In recent decade, many state-of-the-art algorithms on image classification as well as audio classification have achieved noticeable successes with the development of deep convolutional neural network (CNN). However, most of the works only exploit single type of training data. In this paper, we present a study on classifying bird species by exploiting the combination of both visual (images) and audio (sounds) data using CNN, which has been sparsely treated so far. Specifically, we propose CNN-based multimodal learning models in three types of fusion strategies (early, middle, late) to settle the issues of combining training data cross domains. The advantage of our proposed method lies on the fact that We can utilize CNN not only to extract features from image and audio data (spectrogram) but also to combine the features across modalities. In the experiment, we train and evaluate the network structure on a comprehensive CUB-200-2011 standard data set combing our originally collected audio data set with respect to the data species. We observe that a model which utilizes the combination of both data outperforms models trained with only an either type of data. We also show that transfer learning can significantly increase the classification performance.

Introduction
Identifying the species of a bird is a widely-studied problem to ornithologists, and an important task in ecosystem monitoring and biodiversity preservation. Recognition of bird species in images is a challenging task due to various appearances, backgrounds, and environmental changes. Despite of this, this fine-grained recognition task has received a significant amount of attention in the computer vision community [1][2][3][4] because of its potential widespread applications. Compared to generic object recognition, fine-grained recognition benefits more from learning critical parts of the object that can help align objects of same class and discriminate between neighboring classes. Current state-of-the-art methods (e.g., [1,2,[5][6][7]) adopt CNN-based architectures that learn representations directly from the raw data and can be used to extract set of discriminative features.
On the other hand, sound also provide us important information about the world around us. Many animals make sounds either for communication or their living activities such as moving, flying, mating etc. Although sound is in some case complementary to visual information, such as when we listen to something out of view, vision and hearing † Graduate School of Engineering, Department of Design and Media Technology, Iwate University, Iwate, 020-8551 Japan † † Graduate School of Engineering, Information Science, University of Fukui, Fukui, 910-8507 Japan a) E-mail: naranchimeg@scv.cis.iwate-u.ac.jp b) E-mail: zhang@u-fukui.ac.jp c) E-mail: akashi@iwate-u.ac.jp (Corresponding author) are often informative about the same structures in the world [8]. As a consequence, numerous efforts have been devoted to recognize bird species based on auditory data [9,10] in recent years. Adapting CNN architectures for the purpose of audio event detection has become a common practice and generating deep features based on visual representations of audio recordings has proven to be very effective [11] such as in bird sounds [10,12]. To improve the affinity between images and sounds, we utilize the CNN to extract features from spectrogram of audio recordings.
In the real world, human are able to handle information consist of different information from multiple plural modalities. In the filed of machine learning, multimodal recognition can improve performance compared with unimodal recognition by utilizing complementary sources of information [13,14]. Multimodal learning have been used for tasks such as image and sentence matching [15], RGB-D object recognition [16], action detection [17] and specially speech recognition [15,[18][19][20], fusing different modalities. The results have shown that one modality can enhance the performance of the other by providing relevant information. Furthermore, authors of [14,21] proposed fusion schemes for multimodal learning with considering the architectures of neural networks. Our main contributions are summarized as follows: • We propose that the combination of image and sound provide richer training signal for bird species classification under CNN framework, which is the first attempt to the best of our knowledge. • Three strategies are investigated for fusing audio and image modalities using CNN. • We collect at least 10 audio recordings for each bird over 178 species, corresponding to the image dataset CUB-200-2011 [22].
Specifically, we adopt CNN to process jointly the two modalities for bird species classification. Three strategies are investigated for fusing audio and image modalities using CNN: modalities is superior. In addition, we apply a two-stage training procedure, which improves the classification accuracy. The rest of the paper organized as follows. In section 2, we review related works on bird species identification with CNN and multimodal learning algorithms. Section 3 describes several architectures among which a multimodal CNN processing jointly the two modalities. In section 4, we describe the integrated dataset utilized for the evaluation experiments of our architectures. Section 5 describes the experimental setup and results. In section 6, we conclude this paper.

Related work
Our work is related to the problem of recognition from multimodal data as well as convolutional neural networks for image classification. We will briefly highlight connections and differences between our work and existing works.
Deep neural networks (DNN) have successfully applied for single modality such us text [23][24][25], images [26][27][28] and audio [29,30] showing their ability to learn representations directly from raw data and can be used to extract a set of discriminative features. CNN is one powerful deep architecture of DNN commonly utilized for image classification [31][32][33]. The use of CNN for distinguishing between fine grained categorization such as bird species categorization has been proposed in many studies [1,2,[5][6][7], which employ part/pose based approaches to achieve good performance. Besides, CNNs have also been applied for speech processing [34,35]. In [11,36], authors utilized CNNs to extract features from spectral representations of audio recordings. It has proved to be efficient when extracting features based on spectrograms of audio recordings such as bird sound [9,12], thus we used spectrogram representations for audio data. In this paper, instead of employing part/pose based approaches we will focus on integrating raw image an audio using conventional CNN, in order to know which fusion strategy performs best.
Multimodal learning algorithms have been used for tasks such as image sentence matching [37], action recognition [17], RGB-D object recognition [16], and speech recognition (audio-visual speech recognition [18] and visual-only speech recognition [15]). Among the many approaches for multimodal learning, multimodal integration is commonly realized by three different categories of approaches. First, in early fusion approach, feature vectors from multiple modalities are concatenated and transformed to acquire a multimodal feature vector. For example, Ngiam et al. [14] utilized DNN to extract fused representations directly from multimodal signal inputs. Likewise, in middle fusion approach, Huang et al. [21] employed deep belief network (DBN) to combine mid-level features learned by single modality. Lastly, in late fusion approach, outputs of unimodal classifiers are merged to determine a final classification. For example, in RGB-D object recognition, Eitel et al. [16] proposed two separate CNN streams processing RGB and depth data independently are combined with late fusion approach. Therefore, Simonyan et al. [17] proposed two-stream (one stream processing spatial features from RGB image inputs, while the other stream processing temporal features from optical flow inputs) network architecture designed to recognize action for videos. They combined two streams by concatenating features and by averaging prediction scores from two CNNs, respectively. In contrast to these works, we propose simple yet effective concatenation, summation and multiplication based fusion methods with respect to three strategies.

Methodology
Our three multimodal architectures extend conventional CNN for large-scale image classification [31]. Our implementation is based on CaffeNet , and can be treated as [38] a variation of the structure proposed by Krizhevsky et al. [31].

Feature extraction
CNN uses hierarchical features in its processing pipeline. The features from initial layers are primitive while late layers are high-level abstract features made from combinations of lower-level features. The CaffeNet consists of five convolutional layers (with max pooling layers following the first, second and fifth convolution layer) followed by three fully connected (FC) layers and a softmax classifier. Rectified linear unit is applied to every convolutional layer and fully connected layer and local normalization is applied in first and second convolutional layer. The process through this 8-layer CNN network can be treated as a process from low to mid to high-level features. We hypothesize, that combining the features of different layers in this pipeline can lead to achieve better performance.

Feature combination
We propose our method in three strategies to fuse features: early fusion, middle fusion and late fusion. Early fusion, also known as feature level fusion, is a feature combination scheme that features from multiple modalities concatenated to form a merged feature vector. Middle fusion, also called mid-level combination, combines the high-level features learned by single network. Late fusion, also called decision-level fusion, combines the decisions of the unimodal classifier and determine the final classification. In this paper, we use concatenation to combine low and highlevel features, and summation or multiplication to combine decisions of the classifiers. The feature combination layers can be trained with standard back-propagation and stochastic gradient descent.

Architectures of proposed models
The proposed multimodal learning models which combines audio and image using CNN by different fusion approaches . 227 × 227 pixel RGB images of two modalities are concatenated at merging layer, which produces 227 × 454 × 3 output volume, and the convolution layers will extract joint features from this merged volume. We use HDF5 format to manage datasets of two modalities, because of it's flexible data storage and unlimited data types.
are described in this section. We exploit the same architecture for both audio and image modalities to focus on evaluating the effectiveness of the feature combination approaches.

Early fusion model
One direct approach for combining audio and image is to train a CNN over the concatenated audio and image data as shown in Fig. 1. In this strategy, the input vectors related to each modality are concatenated together and then processed together throughout the rest of the CNN pipeline. This model is most computational efficient comparing to the middle and late fusion models, because the number of learnable weight parameters is almost half times less than late fusion model.

HDF5
Image The architecture of middle fusion model (Net2). The activations of the pool5 layers of the two modalities are concatenated at merging layer, and feeding it into the three fully connected layers with softmax at the end.

Middle fusion model
In the middle fusion strategy, unimodal features is extracted independently from audio and images, then combined into a multimodal representation by concatenating the activations of the last pooling layers of two modalities. The multimodal representation then learned in the following fully connected layers. The middle fusion model is shown in Fig.2.

Decision or late fusion model
In contrast to the middle fusion model, extracted unimodal features are separately learned to compute unimodal scores, then these scores are integrated to determine a final scores. The late fusion model consists of two-streams processing audio and image data (green and blue) independently, which are fused after last fully connected layers as shown in Fig.3. Among the various ways of combining CNNs with different modalities, one straightforward way is to add additional fully connected layers to combine the outputs of the each streams as presented in [16]. Instead of using FC layer to combine the two streams, we applied element-wise summation and element-wise multiplication to fuse the outputs of the each streams.

Database
We evaluate our models on the popular fine-grained CUB-200-2011 [22] bird dataset and our originally collected sound dataset from sharing bird sound database Xeno-Canto † . CUB-200-2011 [22] dataset contains 11788 images of 200 species of birds, with each image downscaled to 227 × 227 pixel. Spectro-temporal features (spectrogram) are extracted from audio recordings that we collected from the Xena-Canto to be used as the audio representation. Based on the 200 species of the CUB-200-2011, we try to harvest at least 10 different audio recordings for each species. As a result, audio recordings over 178 species were collected † http://www.xeno-canto.org The spectrogram of 10 seconds duration, which will be fed into the bird classification model. completely (#recordings ≥ 10), audio recordings from 19 species were collected deficiently (0 < #recordings < 10), and 3 species could not be collected. The spectrograms of the audio are obtained by using short-time fourier transform (STFT) over 10 seconds audio frames, windowed with Hanning window (size 512, 50% overlap). The reason is, the sounds of birds are usually contained in a small portion of the frequency range (mostly around 2-8 kHz) as stated in [10], so we only extract features from the range of (0, 10) kHz. In order to focus only sounds produced in the vocal organ of birds (i.e. calls and songs), first we obtained the maximum amplitude of the audio and removed a frame which contains only amplitude less than 1/4 of the maximum amplitude. Finally, the spectrograms are saved as 227×227 pixel color images, and the dataset contains 4807 images of 194 species of birds. Several examples including both images and spectrograms over different bird species are shown in Fig.5. We follow the standard training/test split of CUB-200-2011 dataset suggested in [22]. The sound dataset is split into two halves for training and test set respectively.
The multimodal CNNs are trained in a supervised manner, thus we create integrated dataset by matching two data sets and corresponding labels using HDF5 † † file format. Because the HDF5 file can contain any collection of data entities (images, videos, audio recording, text, etc.,) in a single file, and used to organize heterogeneous collections consisting of very large and complex datasets.

Experiments and Results
The experiments in this section are conducted to evaluate the effectiveness of our proposed architectures (Net1, Net2, Net3). In order to evaluate an advantage of our late fusion approach, we conduct a comparative experiment between Net3 model and two different existing fusion approaches [16,17]. Besides, we have fine-tuned a pretrained model for all the models (including comparative models) in order to improve the performance. To improve the repeatability and only focus on the evaluation of fusion step, we use the well-known CaffeNet model [38] to extract features, train, and fine-tune the CNN with default structure and parameter setting fixed except the base learning rate and batch size. We set the learning rate to 0.001 for single modality and 0.0001 for multimodal learning. The batch size is set to 32 for single modality and 1 for multimodal learning due to limited resources.

Multimodality
The core idea of this paper is to address the integration of the image and audio. Therefore, it is necessary to compare the performance between single-modality (image or sound) and multi-modality (both image and sound: Net1, Net2, Net3). To focus on the evaluation of fusion models, in this experiment, we do not introduce any transfer learning techniques (e.g., fine-tune the pretrained model to help differentiate between gains from the proposed architectures). Table 1 summarizes the results of single modality and proposed models. We can observe that combining two modalities using CNN improves the performance of those only using one modality, extracting features separately from image and audio and fusing them at the late stage performs better with significant gain. Interestingly, the performance of low-level and mid-level fusion models slightly better than the performance of single-modality. One possible reason is because CNN learns features for the predominant modality. In contrast, learning features separately for different modalities results in more independent features, which leads to achieve better performance. Let us mention that the result obtained by multi-modality is different from simply combining the results of two CNNs trained separately. Indeed, the two modalities' parameters are jointly estimated and thus can be mutually influenced. We provide a detailed discussion in following section. Figure 6 plots learning process of single modality (image only and audio only) and Net3 model over learning epochs. It can be observed that Net3 significantly improves the results.

Qualitative result: Single-modality v.s. Multimodality
We perform a qualitative study to analyze effects of multimodal learning models by comparing single-modality and multi-modality networks. First, we select some classes where the multi-modality models provides the correct answer while the single-modality model produces the wrong classification. Then, we study why the multi-modality models provides right answers while the single-modality model failed to produce the correct classification. Figure 7 shows some examples of single-modality vs. multi-modality classification. In the first column, the single-modality models predict the input image and spectrogram as the 'barn shallow' and the 'ring-billed gull' respectively rather than the 'redbellied woodpecker'. However, the multi-modality models are able to predict the right answer, because those models provide joint features of different modalities. We observe that when single-modality model provide right answer for spectrogram, the probability of providing the right answer of multimodality models is higher than when single-modality classificaion is correct for image. In the second column of Fig. 7, single-modality model is misclassified the 'clark nutcracker' image as the 'great grey shrike', where other models provide correct answers. Lastly, the Net3 model is able to provide right answers while the other models provide misclassification on the 'belted kingfisher' (last column).
To better understand the difference between the models, we analyze the feature learned from the each networks by visualizing the filters of the first convolutional layers shown in Fig. 8. We see that each network's filters of different models have similar pattern. Secondly, we see that the singlemodality's filters have more meaningful patterns than the multi-modality filters. As we mentioned before its seems that the learning features separately for different modalities results more independent features. Finally, it can be seen that the filters of early fusion model has combined patterns from both networks, but most filters are similar to the spectrogram network. It reveals that the multimodal model (early fusion and mid-level fusion) learns features for the predominant modality.

Quantitative result: Net3 v.s. existing late fusion approaches
To evaluate the effectiveness of our late fusing approach, we conduct comparative experiments on Net3 with existing late fusing methods presented in [16,17]. The differences between the late fusion approaches is shown in Fig. 9.
• FC7 concat [16]: FC7 layers (green and blue) of each networks are concatenated and merge into the fusion layer, which performs tensor multiplication of two vectors. The resulting fusion vector is then passed through one additional fully-connected layer for classification. This means this fusion methods is a linear combination of pair-wise interactions between two features. However, this method is not suitable when the features are in different sizes. • Averaging scores [17]: Each networks focus on learning features from images and spectrograms, respectively, and the final classification is computed as an average of the softmax scores of the two networks. In this fusion method, they do not consider pair-wise interactions between the features. However, this method is suitable when the model consists of different structured network streams.
In terms of pair-wise interactions, our method is similar to FC7 concat method. However, we fused final output of each networks. Figure 6 plots learning curve of Net3 model with different late fusing methods for each epoch, indicating that averaging the Softmax scores gives lowest performance and the our fusing approach performs best.

Fine-tuning the pretrained model
Combining multi-modalities at the late stage of CNN has proven to be more effective, thus we show additional results with fine-tuning CaffeNet pretrained CNN under Net3 model in this section. One natural idea for fine-tuning is to train the model by initializing both image and audio CNNs with the weights and biases of the first seven layers derived from CaffeNet pretrained network, discarding the last fully connected layer. Instead of last fully connected layer of the pretrained model, we randomly place the initialized new fully connected layer for 200-class bird classification (in our experiment, 194 classes due to the lack of audio dataset). Another method of fine-tuning the model is to train Net3 in two stages. First, training the two stream individually followed by a joint fine-tuning. We train the image and audio CNNs separately, adapting the weights of pretrained model and learn the weights of the new 194-class output layer. After this training, the networks can be used to perform separate classification with respect to each modality. After then, we train an entire model by setting their learning rate to zero and only training the fusion part of the network to freeze the individual stream networks. As shown in Table 2, two stage training is resulted best performance, and it proves that additional modality can improve the performance. We found that the fine-tuning pretrained model to two modalities and training them simultaneously, is significant worse than two stage training. We think the problem relates to the difference between the size of two datasets and batch size of single-modality and multi-modalities. In this experiment, we used net surgery † to fine-tune two different CNN using one pretrained model and to fine-tune two pretrained model into our model.
To confirm that the features extracted from sample image and spectrogram during the fine-tuning were meaningful, we visualized the activations of different layers, especially the fused layers in Net3. The results shown in Fig.10 [17] 70.0 Eitel et al. [16] 72.5 allowed us to confirm the learned features were meaningful and qualitatively resembled the sample image and spectrogram. Moreover, the activations of our fused layer takes advantage by incorporating the features from each streams, when each network failed to produce write answer.

Conclusion
In this paper, we proposed three multimodal CNN architectures in different fusion strategies, which can process jointly the image and audio data for bird classification. Experimental results verified that the two-stream multimodal CNN in late fusion strategy outperforms the others. In addition, we proposed summed fusion method to combine multiple CNNs, which shows better performance comparing against several existing fusion methods. Moreover, with the help of two-stage fine-tuning, our method can be more effective. However, there still exist several drawbacks of our method: (1) Choosing the suitable duration based on the vocal features of birds to be recognized, is essential ingredients of improvement, is missed in our current work. (2) Our method is based on the raw image data, thus part detection and extracting features from pose-normalized regions may improve the classification performance.  Fig. 9: Differences between the late fusion approaches.  Here, the red rectangle shows incorrect answer, the green rectangle shows correct answer of the classification.
As the future work, we aim to apply multiple kernel learning to combine multiple modalities, which is able to learn optimal composite kernel through combining basis kernels constructed from different features of modalities. We are also interested in designing new CNN architectures by increasing the number of modalities.