s2fieldsofstudy:
Computer Science
Physics

text:
Accelerating Spectroscopic Data Processing Using Python and GPUs on NERSC Supercomputers

The Dark Energy Spectroscopic Instrument (DESI) will create the most detailed 3D map of the Universe to date by measuring redshifts in light spectra of over 30 million galaxies. The extraction of 1D spectra from 2D spectrograph traces in the instrument output is one of the main computational bottlenecks of DESI data processing pipeline, which is predominantly implemented in Python. The new Perlmutter supercomputer system at the National Energy Scientific Research and Computing Center (NERSC) will feature over 6,000 NVIDIA A100 GPUs across 1,500 nodes. The new heterogenous CPU-GPU computing capability at NERSC opens the door for improved performance for science applications that are able to leverage the high-throughput computation enabled by GPUs. We have ported the DESI spectral extraction code to run on GPU devices to achieve a 20x improvement in per-node throughput compared to the current state of the art on the CPU-only Haswell partition of the Cori supercomputer system at NERSC.

Introduction
The Dark Energy Spectroscopic Instrument (DESI) experiment is a cosmological redshift survey. The survey will create the most detailed 3D map of the Universe to date, using position and redshift information from over 30 million galaxies. During operation, around 1000 CCD frames per night (30 per exposure) are read out from the instrument and transferred to NERSC for processing and analysis. Each frame contains 500 2D spectrograph traces from galaxies, standard stars (for calibration), or just the sky (for background subtraction). These traces must be extracted from the CCD frames taking into account optical effects from the instrument, telescope, and the Earth's atmosphere. Redshifts are measured from the extracted The data is processed in nearreal time in order to monitor survey progress and update the observing schedule for the following night. Periodically, a complete reprocessing of all data observed to-date is performed and made available as data release to the collaboration and eventually released to the public.
The DESI spectral extraction code is an implementation of the spectro-perfectionism algorithm, described in [BS10]. The process of extracting 1D spectra from 2D spectrograph traces for all 500 targets per frame is computationally intensive and has been the primary focus of optimization efforts for several years ([RTD + 17], [STB19]). The DESI data processing pipeline is predominantly implemented using the Python programming language. A strict requirement from the DESI data processing team is to keep the code implementation in Python.
The existing state of the art implementation utilizes a divide and conquer framework to make spectro-perfectionism algorithm tractable on existing computing hardware, see Figure 1. The code utilizes the Message Passing Interface (MPI) via mpi4py to exploit both multi-core and multi-node parallelism ( [DPS05]). The application uses multidimensional array data structures provided by NumPy along with several linear algebra and special functions from the NumPy and SciPy libraries ( [HMvdW + 20], [VGO + 20]). Several expensive kernels are implemented using Numba just-intime compilation ( [LPS15]). All input and output files are stored on disk using the FITS file format. The application is parallelized by dividing an image into thousands of small patches, performing the extraction on each individual patch in parallel, and stitching the result back together.
This has worked well for CPU-only computing architectures such as the Haswell (Intel Xeon Processor E5-2698 v3) and Knights Landing (Intel Xeon Phi Processor 7250) partitions on the Cori 1 supercomputer at NERSC. The new Perlmutter 2 supercomputer system at NERSC will have a partition of GPU accelerated nodes (AMD EPYC 7763, NVIDIA A100 GPU). The goal of this work is to speed up the DESI experiment's data processing pipeline by porting the spectroscopic extraction step to run on the GPU partition of the Perlmutter supercomputer at NERSC.
In early 2020, the team began reimplementing the existing extraction code specter 3 by reconsidering the problem. The DESI spectral extraction problem is fundamentally an image processing problem which historically have been well-suited to GPUs. However, in many places, the existing CPU version of the code used loops and branching logic rather than vector or matrixbased operations. We performed a significant refactor switching key parts of the analysis to matrix-based operations which would be well suited to massive GPU parallelism. Additionally, the refactor enabled more flexible task partitioning and improved node utilization. From this refactor alone, still running only on the CPU, we obtained 1.6x speedup compared to the original CPU version. From here, we began our GPU implementation.  The goal of the algorithm is to extract spectra from raw telescope output. Here we show the raw telescope output for a single "patch" and the corresponding pieces of the extracted spectra. The divide and conquer strategy used in this application divides an image into roughly 5,000 patches which can be extracted in parallel. The extracted pieces are then stiched back together and written to disk for further processing by the data pipeline.
We describe our iterative approach to porting and optimizing the application using NVIDIA Nsight Systems for performance analysis. We use a combination of CuPy and JIT-compiled CUDA kernels via Numba for GPU-acceleration. In order to maximize use of resources (both CPUs and GPUs), we use MPI via mpi4py and CUDA Multi-Process Service. We discuss the lessons we learned during the course of this work that will help guide future efforts of the team and inform other science teams looking to leverage GPU-acceleration in their Python-based data processing applications. We project that new extraction code gpu_specter 4 running on Perlmutter will achieve a 20x improvement in pernode throughput compared to the current production throughput on Cori Haswell.

GPU Implementation
The existing CPU implementation uses NumPy and SciPy (BLAS and LAPACK) for linear algebra, numba just-in-time compilation for specialized kernels, and mpi4py (MPI) for multi-core and multi-node scaling. The code is parallelized to run on multiple CPU cores and nodes using a Single Program Multiple Data (SPMD) programming pattern enabled by MPI through mpi4py. The structure of the program is illustrated in Figure 2, which highlights the main MPI communication points.
In order to leverage the compute capabilities of GPU devices and adhere to the DESI Python requirement, we decided to use a GPU-accelerated Python library. The main considerations for heterogeneous CPU-GPU computing are to minimize data movement between the CPU host and the GPU device and to feed the GPU large chunks of data that can be processed in parallel. Keeping those considerations in mind, we left rest of the GPU programming details to external libraries. There are many rapidly maturing Python libraries that allow users to write code that will 3. https://github.com/desihub/specter 4. https://github.com/desihub/gpu_specter run on GPU hardware, such as CuPy, pyCUDA, pytorch, JAX, and Numba CUDA. We chose to use CuPy [OUN + 17] and Numba CUDA based on our ability to easily integrate their API with our existing code.
The initial GPU port was implemented by off-loading compute intensive steps of the extraction to the GPU using CuPy in place of NumPy and SciPy. A few custom kernels were also re-implemented using Numba CUDA just-in-time compilation. In many cases, we merely replaced an existing API call from numpy, scipy, or numba.jit with equivalent GPU-accelerated version from cupy, cupyx.scipy, or numba.cuda.jit.
The example code below demonstrates how we integrated cupy, numba.cuda, and the NumPy API: We found that this interopability gave us a lot of flexibility to experiment during development. This achieved our initial goal porting the application to run on GPU hardware. In the following sub-sections, we will discuss the major development milestones that lead to the improved performance of the application on GPUs.

Profiling the Code
As discussed in previous work [STB19], the team found a lot of value using profiling tools such as the cProfile Python module. In this work, we used NVIDIA's NSight Systems to profile the application, identify bottlenecks in performance, and focus optimization efforts. We added CUDA NVTX markers (using the CuPy API) to label regions of our code using descriptions that we would be able to easily identify in the profile viewer. Without these labels, it sometimes difficult to decipher the names of low-level kernels that are called indirectly by our application. We generally used a following command to generate profiles of our application: The nsys profile launches and profiles our application. Usuaully, we disable CPU sampling (--sample=none) and only trace CUDA and NVTX APIs (--trace=cuda,nvtx) to limit noise in the profile output. When using MPI, we add the mpirun or equivalent (srun on NERSC systems) executable with its arguments following the arguments to the nsys profile segment of the command. Similarily, when using the CUDA Multi-Process Service, we include a wrapper shell script that ensures the service is launches and shutdowns from a single process per node. Finally, we specify the executable we wish to profile along with its arguments. The --stats=true option generates a set of useful summary statistics that is printed to stdout. For a more detailed look at runtime performance, it is useful view the generated report file using the NSight Systems GUI.
Nsight Systems provides a zoomable timeline view that allows us to visualize the performance of our code. Using Nsight Systems, we can see the regions of our code that we marked with NVTX wrappers, as well as the lower level memory and kernel operations. In Figure 3, we show a screenshot from an early profile of our GPU port using the NSight Systems GUI. At a high-level, we see that memory transfers and kernel executions, respectively, account for 3% and 97% of the time spent on GPU. From this profile, we identified that approximately 85% of the runtime of the application is spent in the "decorrelate" step of the algorithm. We also discovered an unexpected performance issue near the end patch extraction that we were able to solve using NumPy advanced array indexing. The execution time of the decorrelate method is dominated by the eigenvalue decomposition operations. Profiling also helped identify unexpected performance issues in code regions we did not expect.

Maximizing Node Utilization
We use multiple GPUs in our application via MPI (mpi4py). Since the CPU implementation is already using MPI, minimal refactor was required. Each MPI rank is assigned to a single GPU. Mapping MPI ranks to GPUs can be handled using slurm options (--gpu-bind ), setting environment variables such as CUDA_VISIBLE_DEVICES, or at runtime using the CuPy API (cupy.cuda.Device.use() ). We oversubscribe ranks to GPUs to saturate GPU utilization using CUDA Multi-Process Service (MPS), which allows kernel and memcopy operations from different processes to overlap on the GPU. Some care must be taken to avoid over allocating memory on each device. We use a shell script wrapper to ensure the CUDA MPS control daemon is started by a single process on each node process server before launching our application. At NERSC, we use the following script which references environment variables set by the slurm workload manager.
#!/bin/bash # Example mps-wrapper usage: # > srun -n 2 -c 1 mps-wrapper command arg1 ... export CUDA_MPS_PIPE_DIRECTORY=/tmp/nvidia-mps export CUDA_MPS_LOG_DIRECTORY=/tmp/nvidia-log # Launch MPS from a single rank per node if [ $SLURM_LOCALID -eq 0 ]; then nvidia-cuda-mps-control -d fi # Wait for MPS to start sleep 5 # Run the command "$@" # Quit MPS control daemon before exiting if [ $SLURM_LOCALID -eq 0 ]; then echo quit | nvidia-cuda-mps-control fi In Figure 4, we show how performance scales with the number of GPUs used and the number of MPI ranks per GPU. The solid colored lines indicate the improved performance as we increase the number of GPU used. Different colors represent varying degrees of the number of MPI ranks per GPU. In this case, using 2 MPI ranks per GPU seems to saturate performance and we observe a slight degradation in performance oversubscribing further. We reached the GPU memory limit when attempting to go beyond 4 MPI ranks per GPU. The measurements for the analysis shown here were performed on test node at NERSC using 4 NVIDIA V100 GPUs. The Perlmutter system will use NVIDIA A100 (40GB) GPUs which have more cores and significantly more memory than the V100 (16GB) GPUs. A similar analysis showed that we could go up to 5 MPI ranks per GPU on a test system with A100s. We note that while this configuration maximizes the expected GPU utilization on a Perlmutter with 4 A100 GPUs, the 64-core AMD Milan CPU is only at 31.25% utilization with 20 MPI ranks. Later on, we will discuss one way to utilize a few of these spare CPU cores.

Batching GPU Operations
Earlier, we observed that eigenvalue decomposition accounted for a significant portion of the execution time of our program. In the spectro-perfectionism algorithm, an eigenvalue decomposition is performed on the inverse covariance matrix which is then used to calculate the covariance matrix followed by several smaller eigenvalue decompositions that are performaned on the diagonal blocks of the covariance matrix. Since the small eigenvalue decompositions are performed on independent sub-matrices, we tried "batching" (or "stacking") the operations. We noted the existance of a syevjBatched function in CUDA cuSOLVER library which could perform eigenvalue decomposition on batches of input matrices using a Jacobi eigenvalue solver. This was not immediately available in Python via CuPy but we were able to implement Cython wrappers in CuPy using similar wrappers already present in CuPy as a guide. We submitted our implementation as a pullrequest to the CuPy project on GitHub 5 .
In Figure 5, we show profile snippets of that demonstate the improved performance using the Jacobi eigenvalue solvers from the cuSOLVER library. The execution time of the "decorrelate" method improved by a factor of two.
This inspired us to look for opportunities to use batched operations in our program. We found a significant speedup by refactoring the application to extract spectra from multiple patches in a subbundle using batched array and linear algebra operations. This allowed us to leverage batched Cholesky decomposition and solver operations on the GPU (potrfBatched and potrsBatched in the cuSOLVER library). We contributed cupyx.linalg.posv (named after LAPACK's xPOSV routines) to solve the linear equations A x = b via Cholesky factorization of A, where A is a real symmetric or complex Hermitian positive-definite matrix 6 . Our implementation was essentially a generalization of an existing method cupyx.linalg.invh, which was implemented as the special case where the right-hand side of the equation is the Identity matrix. In Figure 6, we compare the profile timelines before and after implementing batch Cholesky decomposition and solver operations. The runtime for extraction over an entire subbundle of 5 spectra is 3.3 times faster using batched Cholesky operations. 5. https://github.com/cupy/cupy/pull/3488 6. https://github.com/cupy/cupy/pull/4291

Overlapping Compute and IO
At this point, we observed that reading the input data and writing the output results accounted for approximately 25%-30% of the total wall time to process 30 frames from a single exposure in series using a single node. The input data is read by a single MPI rank, transferred to GPU memory, and then broadcast to other MPI ranks using CUDA-aware MPI. After extraction, each MPI rank transfers its results back to CPU memory and the results are gathered to the root MPI rank. The root MPI rank combines the results and writes the output to a FITS file on disk. Using spare CPU cores, we were able to hide most of this IO latency and better utilize the resources available on a node. When there are multiple frames processed per node, the write and read steps between successive frames can be interleaved with computation.
In Figure 7, we demonstrate how a subset of the MPI ranks communicate to achieve this functionality. At a high level, the processing of a single frame can be broken down into 3 distinct phases: read, work, and write. The frames are processed in series, frame one (green) is processed, then frame two (orange), and finally frame (three). Panel a shows the non-overlapping sequence of steps to process 3 frames in series. Panel b shows how the overlapping of IO and computation is orchestrated using two additional MPI ranks, dedicated reader and writer ranks. At the start of the program, the reader rank reads the input data while all worker ranks wait. The reader rank performs some initial preprocessing and sends the data to the root computation rank.
Once the data has been sent, the reader rank begins reading the next frame. After the worker root receives the input data, it performs the work which can involve broadcasting the data to additional worker ranks in the computation group (not shown in the diagram). The result on the root computation rank is then sent to a specially designated writer rank. The computation group ranks move on to processing the next frame which has already been read from disk by a specially designated read rank. Meanwhile, the writer rank finishes writing the previous result and is now waiting to receive the next result.
Overlapping compute and IO in this manner effectively hides the intermediate read and write operations between frames processed serially on a node, reducing the wall time by over 60 seconds and providing a 1.34x speedup in per-node throughput.

Results
Throughout development, we performed a standard benchmark after major feature implementations to track progress over time. For DESI, a useful and practical benchmark of performance is the number of frames that can be processed per node-time on NERSC systems. Specifically, we use the throughput measure frames-per-node-hour (FPNH) as the figure of merit (FoM) for this application. This figure enables DESI to cost out how much data it can process given a fixed allocation of compute resources.
A summary of benchmark results by major feature milestone is shown in Figure 8 and listed in Table 1. The benchmark uses data from a single exposure containing 30 CCD frames. After major feature implementations, we typically perform a scan of hyperparameter values to identify the optimal settings. For example, after the "batch-subbundle" implementation, the optimal number of wavelength bins per patch changed from 50 to 30. The baseline FoM for this application on the Edison and Cori supercomputers is 27.89 FPNH and 40.15 FPNH, respectively. The initial refactor improved the CPU-only performance on Cori Haswell by more than 50%. Our initial GPU port achieved 6.15 FPNH on Cori GPU nodes, an unimpressive mark compared to the baseline CPU benchmarks. Using visual profiling to guide optimization effort, we were able to iterively improve the performance to 362.2 FPNH on Cori GPU nodes.
Since the Perlmutter system is not available at the time of writing, we estimate the expected performance by running the benchmark on an NVIDIA DGX-A100 system. A Perlmutter GPU node will have the same NVIDIA A100 GPUs as the DGX system and the newer AMD Milan CPU compared to the AMD Rome CPU on DGX. The projected FoM for this application on the new Perlmutter supercomputer is 575.25 FPNH, a roughly 20x improvement over the Edison baseline.
Going forward, the team will need to re-evaluate where to refocus optimization efforts. The performance of the spectral extraction step is now comparable to other steps in the DESI data processing pipeline. We are currently evaluating other steps in the DESI pipeline for GPU acceleration. The DESI team may also opt to spend the improved efficiency to perform more compute intensive processing if there is a scientific opportunity.

Conclusion
The rising popularity of heterogenous CPU-GPU computing platforms offers an opportunity for improving the performance of science applications. Adapting scientific Python applications to use GPU devices is relatively seamless due to the community of developers working on GPU-accelerated libraries that provide *. Note that the initial-gpu benchmark only processed a single frame instead of all 30 frames from an expoosure.
Numpy-compatible and SciPy-compataible APIs and, of course, the excellent foundation provided by NumPy and SciPy projects. Profiling tools such as NVIDA Nsight Systems and the cProfile Python module often provide actionable insights to that can focus optimization efforts. Refactoring code to expose parallelism and use more vectorized operations often improves performance on both CPU and GPU computing architectures. For DESI, the transition to GPUs on Perlmutter will shorten the time it takes to process years worth of data from weeks to months down to hours to days. Fig. 7: Overlapping IO and compute. In panel a, we show an example timeline of the root worker MPI rank performing the read, work, and write steps to process 3 frames. In panel b, we show an example timeline of the root worker, read, and write MPI ranks performing the read, work, and write steps along with their inter-communication to process 3 frames.


Chord Embeddings: Analyzing What They Capture and Their Role for Next Chord Prediction and Artist Attribute Prediction

Natural language processing methods have been applied in a variety of music studies, drawing the connection between music and language. In this paper, we expand those approaches by investigating \textit{chord embeddings}, which we apply in two case studies to address two key questions: (1) what musical information do chord embeddings capture?; and (2) how might musical applications benefit from them? In our analysis, we show that they capture similarities between chords that adhere to important relationships described in music theory. In the first case study, we demonstrate that using chord embeddings in a next chord prediction task yields predictions that more closely match those by experienced musicians. In the second case study, we show the potential benefits of using the representations in tasks related to musical stylometrics.

Introduction
Natural language processing (NLP) methods such as classification, parsing, or generation models have been used in many studies on music, drawing the connection that music is often argued to be a form of language. However, while word embeddings are an important piece of almost all modern NLP applications, embeddings over musical notations have not been extensively explored. In this paper, we explore the use of chord embeddings and argue that it is yet another NLP methodology that can benefit the analysis of music as a form of language. Our objectives are (1) to probe embeddings to understand what musical information they capture, and (2) to demonstrate the value of embeddings in two example applications.
Using word2vec [18] to create embeddings over chord progressions, we first perform qualitative analyses of chord similarities captured by the embeddings using Principal Component Analysis (PCA). We then present two case studies on chord embedding applications, first in next-chord prediction, and second in artist attribute prediction. To show their value, we compare models that use chord embeddings with ones using other forms of chord representations.
In the next chord prediction study, we provide a short chord sequence and ask what the next chord should be. We collected human annotations for the task as a point of reference. By comparing model predictions with the human annotations, we observe that models using chord embeddings yield chords that are more similar to the predictions of more experienced musicians. We also measure the system's performance on a larger set drawn from real songs. This task demonstrates a use case for chord embeddings that involves human perception, interaction, and composition. For the artist attribute prediction study, we perform binary classification tasks on artist type (solo performer or group), artist gender (when applicable), and primary country of the artist. Results on these tasks demonstrate that chord embeddings could be used in studies of musical style variations, including numerous studies in musicology.
This paper contributes analyses of the musical semantics captured in chord2vec embeddings and of their benefits to two different computational music applications. We find that the embeddings encode musical relationships that are important in music theory such as the circle-of-fifths and relative major and minor chords. The case studies provide insight into how musical applications may benefit from using chord embeddings in addition to NLP methods that have previously been employed.

Related Work
Methods for learning word embeddings [18,22,23,4] have been useful for domains outside of language (e.g., network analysis [7]). Recent work has explored embeddings for chords, including an adaptation of word2vec [15], their use in a chord progression prediction module of a music generation application [3], and for aiding analysis and visualization of musical concepts in Bach chorales [24]. However, understanding the musical information captured as latent features in the embeddings has been limited by the decision to ground evaluation in language modeling metrics (e.g., perplexity) rather than analyses of their behavior in downstream tasks. In this work, our first case study shows that language models with no remarkable differences in performance by perplexity exhibit remarkable relationships in their predictions to the experience of musicians, and furthermore we provide insights into what is captured by the embeddings.
NLP methods have benefited computational musicology topics such as authorship attribution [30], lyric analysis [6], and music classification tasks using audio and lyrics [16]. In the task of composer identification, many approaches draw inspiration from NLP, applying musical stylometry features and melodic ngrams [2,31,8]. One study used language modeling methods on musical n-grams to perform composer recognition [10] and another used a neural network over encodings of the pitches of musical pieces [12]. Similar methods have been used to study stylistic characteristics of eight jazz composers [19,1] over chord sequences similar to ours. While these studies operated on small datasets (on the order of hundreds of samples) to identify and analyze music of a small set of musicians, we use a large dataset (on the order of tens of thousands) and predict attributes of artists based on the music.
Our attribute prediction tasks are related to NLP work in authorship attribution and are motivated by studies on the connection between language and music in psychology [21,11], and the intersection of society, music, and language, or sociomusicology [5,28]. For instance, Sergeant & Himonides studied the perception of gender in musical composition, and found no significant match between the listener's guess of a composer's gender and their actual gender [27].

Dataset
We compile a dataset of 92,000 crowdsourced chord charts with lyrics from the Ultimate Guitar website. 4 We identify and remove duplicate songs 5 in our data using Jaccard similarity, then extract the chord progressions from our data representation to learn chord embeddings.   We remove songs with fewer than six chords, leaving us with a final set of 88,874 songs and 4,913 unique chords. The chords' song frequencies are distributed according to a power law distribution, much like Zipf's word frequency law exhibited in natural language. Figure 1 shows the song frequency and the power law trend line fitted to the top 61 chords for demonstration, though the trend is even stronger over all chords. Beyond there existing many possible chord notations, we observe a relation between chord frequency and physical difficulty of playing the chord on the guitar or other instruments (e.g., G, C, and D), and variations in notation.

Chord Embeddings
Word embeddings have been used extensively to represent the meaning of words based on their use in a large collection of documents. We consider a similar process for chords, forming representations based on their use in a large collection of songs.
We create chord embeddings for chords that appear in at least 0.1% of our songs (237 chords) using the continuous bag of words model (CE cbow ) and the skip-gram language model (CE sglm ) from word2vec [18], a widely used method for creating word embeddings. For CE cbow , a target chord is predicted based on context chords, while CE sglm is the reverse: context chords are predicted given a target chord. In both cases, this has the effect of learning representations that are more similar for chords that appear in similar contexts. We tested context sizes of two, five, and seven, and varied the vector dimensions between 50, 100, 200, and 300, but observed only minor differences across different models, and chose to use a context window of five and a vector dimension of 200.

Qualitative Analyses
To better understand the information encoded in chord embeddings, we perform a qualitative analysis using PCA and present a 2D projection for the CE sglm model in Figure 4 (our main observations are consistent for the case of CE cbow ).
We observe that chords that form a fifth interval are closer together, which suggests that the embeddings capture an important concept known as the circle of fifths. Fifth-interval relationships serve broad purposes in tonal music (music that adheres to a model of relationships to a central tone). Saker [26] encapsulates their structural importance by stating that "circle of fifths relationships dominate all structural levels of tonal compositions" and that "the strongest, most copious harmonic progressions to be found in tonal music are fifth related." The circle of fifths relationship is observed in our chord embeddings over different chord qualities, 6 specifically over major chords (highlighted in Figure 4b), minor chords (highlighted in Figure 4c), major-minor 7 chords, and minor 7 chords. For both chord qualities, the layout captured by the chord embeddings is similar to the ideal/theoretical circle of fifths, illustrated in Figure 4a. This pattern is particularly interesting as it does not follow the style of word analogy patterns observed in language. This makes sense, as the "is-a-fifth" relation forms a circle in chords, whereas word analogies connect pairs of words without forming a circle. Additionally, we observe that relative major and minor chords 7 appear relatively close together in the embedding space, as shown by their proximity in the PCA plots (highlighted in Figure 4d). We also observe that enharmonics, notes with different names but the same pitch, are often close together. Not only that, but there is a consistent pattern in the positioning of enharmonics, with sharps to the left and flats to the right.
These observations suggest that chord embeddings are capable of representing musical relationships that are important to music theory. Transitions between the tonic (I), dominant (V), and subdominant (IV) chords of a scale are prescriptive components in musical cadences [25]. Since these chords frequently appear in the same context, their embeddings are more similar. A common deceptive cadence is a transition between the fifth and sixth root chords of a scale [20]. An example progression with a deceptive cadence in C major is C-F-G-Am; these chords appear in a similar neighborhood in the PCA plots. Because these chords frequently co-occur in music, the embeddings capture a relationship between them.
We also note that relationships for chords that are used more frequently are more strongly represented. The major and minor relative pairs (G, Em), (C, Am), and (D, Bm), are among the top ten chords ranked by song frequency (Figure 1) and have clear relations in Figure 4. In contrast, the pairs (Ab, Fm) and (Db, Bbm) are ranked lower and their minor-major relative relationship appears weaker by their distance.

Alternative Representations
In addition to chord embeddings, we also explore two other chord representations: Pitch Representations (PR) and Bag-of-Chords (BOC). For a fair comparison, we use the same vocabulary of 237 chords for these representations. Pitch Representations. A chord's identity is determined by its pitches, so we test if the individual pitches provide a better representation of a chord as a whole than our chord embeddings. This method represents each chord by encoding each of its pitches by their index in the chromatic scale {C = 1, C# = 2, · · · , B = 12}. The pitches are in order of the triad, followed by an additional pitch if marked, and by one extra dimension for special cases. 8 Additional pitches are indicated by the interval relative to the root of the pitch that is being added. We also represent chord inversions, e.g., the chord G/B which is an inversion of G such that the bottom pitch is B. Bag-of-Chords. In this representation, each chord is represented as a "one-hot" vector, where the vectors have length equal to the vocabulary size. We consider two ways of determining the value for a chord. For BOC count , we use the frequency of a chord in a song, divided by the number of chords in the song. For BOC tf idf , we use the TF-IDF of each chord (term-frequency, inverse document frequency).

Case Study One: Next Chord Prediction
In this section, we present our first case study, which investigates if there is a relationship between chord embedding representations and the ways humans 8 Special cases include: the "*" marking on a chord, which is a special marker specific to the ultimate-guitar.com site; "UNK" which we use to replace chords that do not meet the 0.1% document frequency threshold; and "H" and "Hm" which indicates "hammer-ons" in the notation on ultimate-guitar.com perceive and interact with chords. We test the use of chord representations for predicting the most likely next chord following a given sequence of chords, and then compare these to human-annotated responses.
We train a next chord prediction model using a long short-term memory model [9] (LSTM). 9 We follow standard practice and do not freeze the embeddings, meaning the chord representations undergo updates throughout training, adjusting to capture the musical features most important for the task. Our main model uses the pre-trained chord embeddings to initialize the chord prediction architecture. We test both CE cbow and CE sglm embeddings, and will refer to these models by these acronyms. We also define a baseline model, where the encoder is randomly initialized (denoted NI, for no initialization). Finally, we also evaluate a model where we initialize the encoder with the pitch representations introduced in Section 4.2 (denoted PR).

Human Annotations
To evaluate the next chord prediction models, we collect data with a human annotation task in which annotators are asked to add a new chord at the end of a chord progression. For example, given the progression "A, D, E," they must pick a chord that would immediately follow E. They are also asked to pick one or two alternatives to this selection. Continuing the example, if an annotator provides E7 and A, then the chord progressions they have specified are "A, D, E, E7," and "A, D, E, A." The annotators are given a tool to play 48 different chords (all major, minor, major-minor 7, and minor 7 chords) so they can hear how different options would sound as part of the progression. 10 They were given a total of 39 samples shown in the same order to all annotators. The samples were chosen randomly from our entire dataset of songs, permitting only one sequence to come from a single song, and requiring they contain the same 48 chords. We presented sequences of length three and six as we expect that patterns in the given sequence affect the responses.
Participants. The annotators were first asked to estimate their expertise in music theory on a scale from 0 -100, where 0 indicates no knowledge of music theory, 25 -75 indicates some level of knowledge from pre-university training though self-teaching, private lessons/tutoring, or classroom settings, and 75 -100 indicates substantial expertise gained by formal university studies, performing and/or composing experience. They were given the option to provide comments about how they estimated their expertise. We collected this information because we expected that the annotations provided by a participant may vary depending on their background education in music theory. It also allows us to perform comparisons of our system with sub-groups defined by self-reported knowledge. Nine participants provided complete responses, with expertise ratings of 0, 0, 10, 10, 19, 25, 25, 50 and 73. For the following analyses, we define a beginner set containing annotators who provided 0 for their self-rating, an intermediate set containing annotators with ratings > 0 and < 50, and an expert set containing annotators whose ratings are at least 50.

Evaluation Metrics
The main objective of this case study is to investigate whether chord similarities captured by our embeddings reflect human-perceived similarities. We use the chord-prediction systems to perform the same task given to the annotators. Each model provides a probability distribution over the full set of chords, therefore we treat the chords with highest probabilities as each model's selection.
We evaluate the predictions with the following metrics, which are inspired by the metrics employed by the Lexical Substitution task [17] but modified for our setup, which weight more frequent responses higher: Pitch Matches: The metrics above penalise all differences in predictions equally, even though some chord pairs are more different than others, e.g., A7 and A differ only in the addition of a pitch, whereas B and A share no pitches. To address this, we use a metric that is the total number of pitches that match for each question between the model's top response and the annotator's first response. We calculate this separately per-annotator, and then average across annotators (P M ave ).
Loss and P erplexity(P P L): These are two measures from the language modeling literature that we apply to see how well the models do on the true content of songs. Note that this evaluation is over a different set: 8,748 randomly chosen songs that are not included in model training.

Results
Match and Mode. Table 1 reports metrics for each system when compared with the beginner set, intermediate set, expert set, and full set of annotators. When evaluating against all annotators, CE sglm is best in M atch best and M atch oo4 , NI is best in M ode best , and PR is best in M ode oo4 .
In the results for subsets of annotators, all systems tend to match experts better than the beginner or intermediate groups. In particular, CE cbow obtains the lowest scores when evaluated against the beginner group, but the highest when evaluated against the expert group. To investigate this pattern, we compute M atch best and M atch oo4 for each individual annotator and then the Spearman Correlation Coefficient r s between these metrics and their expertise. In Table 2, we observe strong significant correlation between the CE cbow and expertise, and no significant correlation for the other models. This can be explained by the fact that the models are trained on a large collection of songs composed by experts, and the chord embeddings seem to capture the chord use style of the experts. Pitch Matches. We report P M ave for the models and expertise groups in Table 1. Similarly to the M atch and M ode metrics, we observe that all models perform better when compared to annotators with higher expertise, and the differences between groups is most extreme with CE cbow . Table 2 shows the results of correlation analysis for pitch matches (Pearson Correlation Coefficient r p ), with a significant linear correlation for only CE cbow . This trend is visually shown in Figure 3.   Table 3. Loss and perplexity metrics for the chord prediction models on a held-out test set.
Loss and Perplexity. Table 3 shows the results on our 8,748 song test set. All models perform similarly in this setting.

Discussion
We observe that automatic models produce predictions that resemble human responses. Comparing against annotators grouped by expertise, CE cbow compares best to the high expertise group for all metrics, while the best model varies among the other groups. CE cbow 's predictions also correlate significantly with pitch matches and annotator expertise. NI and PR achieve the highest M ode best and M ode oo4 scores, however, fewer samples are considered because only twentyfive had a unique mode chord across the annotators and only five of these samples had more than half the annotators agree. Additionally, the chord symbol based metrics are strict, requiring an exact match on chords, and had lower interannotator agreement than the pitch-based metrics. While CE cbow 's predictions exhibit a strong pitch-match correlation, CE sglm 's predictions exhibit no significant correlate at all. However, differences between the CE cbow and CE sglm embeddings may not be as apparent in other downstream applications; in fact, by the perplexity and test loss metrics shown in Table 3, there is barely a difference between these two, or any, models. Investigating key differences between these embedding models in musical contexts is a direction for future work.

Case Study Two: Artist Attribute Prediction
Our second case study introduces the task of performing artist attribute prediction, demonstrating that these chord representations could be used more broadly in tasks involving musical stylometry and musical author profiling. With binary classifiers using our chord representations, we predict three attributes as separate tasks: gender (male or female), performing country (U.S. or U.K.), and type of artist (group or solo artist).
Data. For these experiments, we augment the dataset with information obtained with the MusicBrainz API, 11 which includes the song artist's location, gender, lifespan, tags that convey musical genres, and other available information for 35,351 English songs (identified using Google's Language Detection project [29]). From this extracted information, we choose artist type, performing country, and gender because of the sufficient quantity of data available with these attributes enabling the tasks; we note that tasks dedicated to genre or time period are of interest for future investigations, and our preliminary experiments using the artists' lifespan and tags as proxies for time period and genre indicated these tasks are promising use cases for chord embeddings.
We use the top two most frequent classes of each attribute, and balance the data to have the same number of examples for each class. For artist type, there are 20,000 songs per class (group and solo). For performing country, there are 8,000 songs per class (U.S. and U.K.). For gender, there are 6,000 songs per class (male and female). The number of samples varies because of differences in the raw class counts and because not all songs have a label for each property.
Experimental Setup. We build two binary classifiers and compare their performance with each chord representation. The first uses logistic regression (LR) over a single vector for each song by aggregating chord representations. We also experimented with an SVM classifier, but LR was more efficient with minimal performance trade-offs. The BOC methods are defined in Section 4.2. The PR method aggregates the chords with a many-hot encoding vector counting each chord pitch, normalized by the total number of pitches. The CE methods aggregate chord embeddings by max-pooling, taking the most extreme absolute value in each dimension across all chords.
The second classifier is a Convolutional Neural Network (CNN) that considers the chords in sequences. We experimented with an LSTM, and found that the CNN functions better for these tasks. We use the CNN model for sentence classification by Kim [13] 12 over the chord progressions for each song, using the same NI, PR, CE sglm , and CE cbow representations from the first case study.
For our model parameters, chosen in preliminary experiments on a subset of the data, we use L2 regularization for the LR classifier, and the CNN model uses filter window sizes 3, 4, 5 with 30 feature maps, drop-out rate 0.5 and Adam [14] optimization. The sequence limit is 60 chords, cutting off extra chords and padding when there are fewer.

Model
Gender Country Artist Type  Table 4. Accuracy scores from 10-fold cross validation in artist gender, country, and type prediction tasks. The significance tests are performed among the logistic regression models and CNN models separately. Table 4 shows the models' accuracy scores from experiments using 10-fold cross validation. CE sglm CNN is the top performer for all tasks, significantly outperforming CNN PR and all LR models for all tasks, and NI for country. 13 All models outperform a random baseline (50%) significantly in each task. In each task, CNN models significantly outperform their LR counterparts. For insight into the models' performance, we analyze the gender prediction task, the only attribute where the LR CE cbow and LR BOC count predictions differed 12 CNN model is built on https://github.com/Shawn1993/ cnn-text-classification-pytorch 13 By a paired t-test with p < .05.

Results and Analyses
significantly. First, we compare the rate of use of each chord between genders. To show the differences in Figure 4a, we divide the higher rate by the lower rate, subtract one to set equal use to zero, and flip the sign when female use is higher. We observe greater variations among chords with lower song frequency. For instance, C/G, F#7, Bbm, and Ab are twice as salient for one gender than the other. The highest variation among the top 20 chords reaches 1.5 times more salient, and for the top 10, 1.2 times more salient for one gender.
To investigate the impacts of the musical relationships captured in embeddings (Section 4) to the CE models, we also compared use of five chord qualities. Figure 4b shows higher relative frequency of suspended and diminished chords among the songs of male artists, augmented and minor chords among the female artist songs, and fairly similar use of major chords.

Discussion
To our knowledge, this is the first time that an association between chord representations and author attributes has been explored. Each model for each attribute showed significant improvement over a random baseline of 50%, indicating there are quantifiable differences in our music data between the genders, countries, and artist types. In addition to the tasks we presented, we also observed improvement when using the system to predict the life-period of the artist and their associated genres. As life-period is a proxy for the music's time period, chord embeddings could benefit future work in musical historical analysis. For gender, the variation of rare chords may contribute to BOC count outperforming CE cbow . However, CE cbow significantly outperforms BOC tf idf which gives more weight to rare chords by their inverse-document frequency. This suggests that chord rarity is not the only critical feature. The variations of chord quality use may contribute to the performance of the CE models as the embeddings capture musical relationships.
LR PR consistently underperforms all other models, which may indicate the importance chord structures. Different chords with the same pitches (e.g., G and G/B) have the same PR vector. Chords with overlapping pitches have similar PR vectors. However, CNN PR, which performs closer to the others, encodes pitch orderings (Section 4.2) and BOC methods encode chord symbols which indicate structure. CE representations are learned from chord symbols, likely capturing contextual functions of chord structures. These functions would matter for the CNN models which make predictions from chord sequences rather than a single aggregated vector. Since we observed the best performance by CNN CE sglm , there suggests the importance of contextual semantics of chord structures. A deeper study into structural semantics captured by chord embeddings is a direction for future work.

Conclusion
In this paper, we presented an analysis of the information captured by chord embeddings and explored how they can be applied in two case studies. We found that chord embeddings capture chord similarities that are consistent with important musical relationships described in music theory. Our case studies showed that the embeddings are beneficial when integrated in models for downstream computational music tasks. Together, these results indicate that chord embeddings are another useful NLP tool for musical studies. The code to train chord embeddings and the resulting embeddings, as well as the next-chord annotations are publicly available from https://lit.eecs.umich.edu/downloads.html.


extfieldsofstudy:
Computer Science
Engineering

text:
Cascaded encoders for unifying streaming and non-streaming ASR

End-to-end (E2E) automatic speech recognition (ASR) models, by now, have shown competitive performance on several benchmarks. These models are structured to either operate in streaming or non-streaming mode. This work presents cascaded encoders for building a single E2E ASR model that can operate in both these modes simultaneously. The proposed model consists of streaming and non-streaming encoders. Input features are first processed by the streaming encoder; the non-streaming encoder operates exclusively on the output of the streaming encoder. A single decoder then learns to decode either using the output of the streaming or the non-streaming encoder. Results show that this model achieves similar word error rates (WER) as a standalone streaming model when operating in streaming mode, and obtains 10% -- 27% relative improvement when operating in non-streaming mode. Our results also show that the proposed approach outperforms existing E2E two-pass models, especially on long-form speech.

INTRODUCTION
End-to-end (E2E) automatic speech recognition systems (ASR) have made tremendous progress over the last few years, achieving word error rates (WER) that match or surpass conventional ASR models in several common benchmarks [1,2,3,4]. Typical E2E systems consist of a single neural network that transforms input audio to sequences of output tokens, like characters or word-pieces, that can be readily transformed to the final sequence of words. Examples of such models include connectionist temporal classification (CTC) [5], attention-based encoder-decoder models [6] like listen-attendspell (LAS) [7], recurrent neural net transducer (RNN-T) [8], and other interesting variations [9,10,11].
The architecture of E2E model, and importantly that of the encoder, partly depends on the target application that the model will be used for. For instance, a number of applications that involve enduser interaction, like voice-search or on-device dictation, require the model to perform recognition in a streaming fashion 1 . As a result, models that use future context to improve performance, like the ones that use bi-directional LSTMs, are not suitable. On the other hand, applications like offline video captioning do not require the model to be streaming and can make use of future context to improve performance. Since these requirements are task specific, separate models are usually trained for streaming and non-streaming applications.
For streaming applications, two-pass models have been proposed to make-up for the performance loss due to lack of future context [12,13]. Similar to a conventional second pass rescoring model [14], these models are used to rescore hypotheses generated by the streaming first pass model. For example, Sainath et al. [12] use an LAS decoder that attends on the output of the encoder of a streaming RNN-T model. Hu et al. extend this work by using a deliberation model [15] that simultaneously attends on the output of the encoder and the first-pass hypotheses. Apart from rescoring the first pass hypotheses, the second pass models can also be used to re-decode the input at the expense of additional computation cost.
One of the shortcomings of LAS decoder based two-pass models is that they suffer from long-form issues -the word deletion rate of an LAS decoder is higher compared to an RNN-T decoder on longform speech [16]. Therefore, LAS-based two-pass models are not applicable in long-form conditions, as we show in our experiments. The model that we propose, called cascaded encoders, alleviates this problem, and enables the same model to be used for streaming and non-streaming applications with improved performance in all conditions. It does that by replacing the second pass LAS decoder with an RNN-T decoder that is shared with the first pass model. Cascaded encoders is motivated by the observation that a single model can behave differently based on the input it receives [17,18]. Shi et al. [18] propose a multi-talker ASR model that predicts targets in an iterative fashion by changing the features that are input to a shared model. Dabre and Fujita [17] use the same parameters for all layers of an encoder, and show that the recurrence depth can be dynamically varied during inference. The main observation is that a single model can learn to map different input distributions to the same or different targets, if they have been exposed to these mappings during training. The work on multi-domain models for ASR [19] and domain (or dialect) identifier-based multidomain [3] and multilingual [20] models also exploit this behaviour.
With cascaded encoders, we show that a single RNN-T decoder can learn the distribution of higher-level acoustic features generated by both streaming and non-streaming encoders of an E2E model. Furthermore, we show that the non-streaming encoder can be trained directly on the output of the streaming encoder instead of input acoustic features; this allows the non-streaming decoder to use fewer layers as opposed to a fully non-streaming model, with a very limited loss in performance. The idea of cascaded encoders is most similar to the recent works on universal ASR [21] and transformertransducer [22] for unifying streaming and non-streaming models. While these works focus on specific encoder architectures (conformer [2] and context-net [23] in [21], and tranformers in [22]), the current work shows that any pairing of encoders can be used to unify streaming and non-streaming use-cases.
The rest of the paper is organized as follows. In Sec. 2 we discuss the model architecture and the methodology for training cascaded encoders. We present our experimental setup in Sec. 3. Sec. 4 presents results and comparisons that show performance using various combinations of architectures, and in a latency constrained setting. Conclusions are presented in Sec. 5.

MODEL DESCRIPTION
The block diagram of the proposed model is shown in Fig. 1. The model is based on the recurrent neural net transducer (RNN-T), which consists of an encoder and a decoder. As input, the encoder of an RNN-T receives acoustic features, x = [x1, · · · , xT x ], where xt ∈ R d corresponds to the input at time frame, t. Unlike attention based E2E models, like LAS [7], the RNN-T decoder has a prediction network, typically an LSTM, that directly models the label sequence y = [y1, · · · , yR], where yr ∈ Y, the output token vocabulary. The prediction network processes labels independent of the acoustics; the RNN-T decoder uses a joint network to combine acoustic encoder's output and the prediction network's output to make the final prediction.

Cascaded Encoders
The encoder layers of an RNN-T, typically, is either causal, like a uni-directional LSTM, or non-causal, like a bidirectional LSTM 2 . But the proposed cascaded encoders model consists of both causal and non-causal layers. The input features, x, are first passed to a causal encoder, which transforms the features to a higher-level representation, e s = [e s 1 , · · · , e s Te ]. The non-causal encoder, which is connected in cascade to the causal encoder, receives e s as input, and outputs e a = [e a 1 , · · · , e a Te ]. Both the causal and the non-causal encoders are directly connected to a shared RNN-T decoder.
Since there are 2 input processing paths -[casual-encoder → RNN-T decoder] and [causal-encoder → non-causal-encoder → RNN-T decoder] -the model's loss consist of 2 components: (1) where Ls and La are the losses from the causal and the non-causal connections to the decoder, respectively. The total loss can be computed as the weighted sum of these components: where λ is a weighting term. In practice, we found that we can decrease the step-time during training by sampling from e s and e a within a mini-batch using λ as the sampling rate. Therefore, for each input utterance, we stochastically choose either the causal or the non-causal processing path at each training step. This alleviates the need to compute RNN-T loss twice for each training example at each training step. With sampling, the model converges roughly after the same number of steps as a standalone streaming model. During inference, the model can operate either in causal or noncausal mode, depending on the output encoder from which the features are fetched for decoding.
Cascaded encoders offer a few advantages over the LAS-based two-pass models. Training such models, typically, is a two-stage process, since the first and the second pass models are trained independently. But cascaded encoders can be trained in a single stage, simplifying the overall process. The model also use a shared RNN-T decoder, which is better suited to handle long-form speech than LAS [16]: LAS-based second pass models do not provide improvements over the first pass model in long-form conditions; it can only operate in rescoring mode when used with long-form audio. Cascaded encoders, on the other hand, can provide gains in all conditions, as we show in Sec. 4. A potential disadvantage of cascaded encoders is that it is more expensive to use the non-causal layers to do rescoring compared to an LAS-based model, because the RNN-T decoder has to marginalize over all possible alignments to compute the posterior of an output sequence that needs rescoring.

Encoder architectures
Apart from the widely used LSTM model, we also present results using the conformer architecture [2] in Sec. 4. Conformer combines the best of transformer [24] and context-net models [23]; it uses both multi-head self attention and convolutional modules within each layer to efficiently combine local and global information. Since conformer is based on transformers, it is also straightforward to limit the amount of right context used in each conformer layer, and hence the entire model, unlike the limited-context LSTM models [25,26]. This can be done my masking the attention score so that only a window of contiguous frames are used to compute the output, instead of the entire input [10]. The same strategy can be used to replace LSTMs with a fully causal conformer encoder in an RNN-T model, and referred to as conformer-transducer (C-T) in this work, to differentiate it from RNN-T.
In this work, we examine using conformer layers to implement either the causal encoder or the non-causal encoder or both, and show that encoders can be cascaded irrespective of the architecture adopted to implement the causal and non-causal layers of the model.

FastEmit
For streaming applications, it important to make predictions with minimum per-word and endpointer latency. Merely using a causal encoder is not always sufficient to meet these constraints [27]. Additional penalties are typically added to the RNN-T loss to improve overall latency, at the expense of increased WER. In this work, we explore whether the losses in WER as a result of adding latency penalties can be recouped using cascaded encoders.
We use the recently proposed FaseEmit loss to improve overall latency (L f e ) [28]. With this, the total loss of the model is: where, β is a tunable hyper-parameter.
stands for element-wise multiplication and 1 causal is an indicator function that is 1 only when the input features to the decoder comes from the causal encoder; L f e is applied only when the decoder is trained in the causal mode since latency is important mainly for the streaming use-case. When optimizing the model to improve overall latency, we also use minimum WER training [29] as it was found to provide better WERlatency trade-offs [27].

Datasets
Our models are trained on the multidomain training sets described in greater detail in [19,30]. The set consists of anonymized and handtranscribed English utterances that are representative of Google's traffic, including handheld and far-field use cases, telephony speech, and YouTube. The data also includes accented speech from multiple English locales [3]. The total amount of training data is ∼400k hours. We augment this training set by simulating noisy conditions at signal-to-noise ratio (SNR) between 0 and 30 dB, with the average SNR of 12 dB, and reverberation times (T 60) between 0 and 900 msec, with an average T 60 time of 500 msec [31]. We also resample input utterances to have roughly equal number of 8k Hz and 16k Hz utterances. SpecAug [1] and random state sampling [30] are used as regularizations.
As evaluation sets, we use short-form and long-form test sets. As short-form test sets, we use ∼12k anonymized and hand-transcribed utterances that represent Google voice-search (VS) and non-voicesearch (NVS) traffic. NVS is relatively simpler than VS, since it has fewer rare-words, but contains utterances that are, on average, twice as long as those in VS. To create the long-form test set, passages containing at least 200 words from a novel are synthesized using a parametric text-to-speech system [32] (T-AB). T-AB contains ∼400 utterances, with an average duration of 65 seconds.

Model architecture
We use domain-id based multidomain models [3] in this work. As features, 128 dimensional log-Mel features are computed on a 32 msec window, with 22 msec overlap between neighboring windows. Features from 4 contiguous frames are stacked to form a 512 dimensional input feature, which is then sub-sampled by a factor of 3 along the time axis. A 16 dimensional one-hot domain-id vector is appended to this to form a 528 dimensional feature, which forms the input to the ASR model. We use multiple model architectures to show that cascaded encoders can be used irrespective of the underlying encoder type. The baseline model is an LSTM-based streaming RNN-T model. It consists of 8 encoder layers of unidirectional LSTMs. Each layer has 2048 cells, which are projected down to 640 output units. A timereduction layer is used after the 2 nd LSTM layer; the features output by the final layer of the streaming encoder is at 60 msec frame rate. The RNN-T decoder consists of 2 LSTM layers in the projection network. Similar to the encoder, each layer has 2048 units that is projected down to 640 output units. The output of the encoder and the projection network is passed to the joint network, which is a 1layer neural net with 640 units. As output representation, we use a word-piece model with a vocabulary size of 4096 [33], and an embedding size of 128. In total, the model has ∼120M parameters When using conformers as the causal encoder in C-T, we use 17 layers. Each layer has 512-units, and uses 8 attention heads and a convolutional kernel of size 15. Note that all attentions are causal in C-T. Similar to RNN-T, the output features are stacked and subsampled by a factor of 2 after the 4 th layer. The decoder still uses an LSTM-based prediction network and has the same structure as the baseline RNN-T model. The C-T model has ∼140M parameters.
All of our models are trained using Tensoflow [34] and the Lingvo toolkit [35], on 8x8 slices of Tensor Processing Units [36]; we use a batch size of 4096 and Adam optimizer [37].

RESULTS
We first present results when using LSTMs for building causal and non-causal layers. So in the subsequent subsection, we show how conformer layers can be used to limit the amount of future context the model has access to. Finally, we show how cascaded encoders can be used to improve performance when there are latency constraints for the first pass model.  Table 1. WERs for baseline (RNN-T) and cascaded encoders RNN-T (CASE RNN-T) models. C and NC stand for causal and noncausal modes of CASE RNN-T. Also shown are results using bidirectional RNN-T, and LAS and LAS-deliberation two-pass models used to decode or rescore hypotheses.

Cascaded Encoders RNN-T model
Tab. 1 shows results using the RNN-T baseline, and the model that uses LSTM-based cascaded encoders (CASE RNN-T). The causal encoder in CASE RNN-T model has the same architecture as the baseline RNN-T. The non-causal encoder is a 2-layer bidirectional LSTM, with ∼10M parameters. Compared to the baseline RNN-T, CASE RNN-T in causal mode obtains very similar WERs. But when used in the non-causal mode, it improves WER on VS by 14% and NVS by 19%. On the long-form T-AB set, WER improves by 16%. A fully bidirectional RNN-T model obtains similar WERs as the non-causal CASE RNN-T on NVS and T-AB sets, but improves VS WER by another 6% relative. The results show that even with 2 layers of bidirectional LSTMs we can obtain WERs that are almost as good a fully bidirectional model using cascaded encoders.
The table also shows results using LAS-based [12] (LAS) and deliberation-based (LAS-Delib) [13] two-pass models models. Performance on VS improves compared to the RNN-T baseline, when using LAS or LAS-Delib in decode mode or rescore mode (first-pass hypotheses generated by RNN-T are rescored in this mode). But only LAS-Delib outperforms CASE RNN-T by 4%, when the latter is used in non-causal mode. And on the slightly longer NVS set, and the long-form T-AB set, non-causal CASE RNN-T outperforms LAS and LAS-Delib. As expected, on T-AB, LAS and LAS-Delib marginally worsens WERs when used in rescore mode, and performs poorly when used in decode-mode. The results clearly show the advantages of CASE RNN-T over LAS and LAS-Delib.  Table 2. WERs for RNN-T, C-T, cascaded encoders RNN-T when using conformer layers in the non-causal encoder (C-CASE RNN-T) or in both causal and non-causal encoders (CASE C-T). For C-CASE RNN-T, also included are WERs when the number of additional conformer layers and the amount of right context are varied. C stands for causal, and NC for non-causal modes.

Limited right context
In order to limit the amount of future context the model sees, we use conformer layers in the non-causal encoder, while continuing to use LSTMs in the causal encoder (C-CASE RNN-T). Each conformer layer, in this setting, has 640 units to match the LSTM layers, and adds ∼10M additional parameters. The results are shown in Tab. 2. Switching the non-causal LSTM layers with conformer layers affect WERs of the causal cascaded encoders model only marginally. Running the same model in non-causal mode yields better WER improvements compared to using bidirectional LSTMs. Using 2 additional layers of conformers with 5.0 seconds of right context yields a relative WER improvement of 19% on VS, 22% on NVS, and 16% improvement on T-AB. Using 3 layers with just 2.9 seconds of right context gives 20% improvement on VS, 19% on NVS, and 20% improvement on T-AB. If we further increase the number of conformer layers to 4, and give the model 15.8 seconds of right context, VS, NVS and T-AB WERs improve by 25%, 25% and 27%, respectively. Also shown in Tab. 2 are results when a conformer based encoder is also used for the streaming encoder. In line with the results in prior work [2], C-T improves over RNN-T by 5% on VS, 3% on NVS and 11% on T-AB. When using conformer-based cascaded encoders (CASE C-T) and decoding in the causal mode, performance on VS degrades by 4% relative on VS, and similar WERs are obtained on the remaining sets. In the non-causal mode, CASE C-T obtains relative improvements of 16% on VS, 13% on NVS and 10% on T-AB compared to C-T. Overall, conformer-based cascaded encoders outperform LSTM-based models in both causal and noncausal mode.
These results, together with those in the previous subsection, show that we can easily combine multiple architectures for causal and non-causal layers in cascaded encoders model, thereby providing a flexible framework for building a unified streaming and nonstreaming model.

Faster first-pass
Tab. 3 shows results when the causal encoder in C-T is trained with FastEmit for improved partial latency, as described in Sec. 2.3. Apart from WER, the table also shows the following latency metrics on the VS test set: 90 th percentile latency from end-of-speech to final endpointing (EP90) and 90 th percentile latency from end-of-speech to first correct partial hypothesis (PR90). PR90 measures how quickly the model predicts the correct hypothesis. As shown, when the cascaded encoders model (CASE C-T) is run in causal model, WER deteriorates on VS by 2%, PR90 worsens by only 30 msec, and EP90 remains unchanged. When run in the non-causal model, which uses 2 additional conformer layers with 5 seconds of right context, WER improves by 21% relative. The results show that the cascaded encoders can be used in a setting where streaming ASR is optimized for low latency, but the final result is generated via the non-streaming encoder that provides large performance gains over the causal model.

CONCLUSION
In this work, we presented cascaded encoders -a flexible framework for combining streaming and non-streaming encoder architecture.
Together with an RNN-T decoder, cascaded encoders can be used to build a single model that can operate in streaming and non-streaming mode. Compared to LAS-based two-pass models that have been proposed in the past, cascaded encoders provides gains for both shortform and long-form speech, and has a simplified one-stage training process. Our results also show that when the first pass model is optimized to minimize the E2E model's latency, cascaded encoders used as a second pass non-causal model can recover the resulting loss in WERs of the first-pass model. Future work will focus on combining cascaded encoders and deliberation models, since they, respectively, provide future context in partly complimentary acoustic and label spaces. Another interesting direction is to explore strategies for speeding up cascaded encoders when they are used as second pass model -either by using rescoring instead of decoding, or by using decoders that use fewer computes than a typical LSTM-based RNN-T decoder.

ACKNOWLEDGEMENTS
We thank Anmol Gulati, Bo Li, Ke Hu, Wei Han, and Yanzhang He for useful discussions and suggestions.