Upload MOSS-TTSD NF4 quantized model

.gitattributes

@@ -33,3 +33,27 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+assets/VS_Open-Source_Models.jpg filter=lfs diff=lfs merge=lfs -text
+assets/VS_Proprietary_Models.png filter=lfs diff=lfs merge=lfs -text
+assets/arch_moss_audio_tokenizer.png filter=lfs diff=lfs merge=lfs -text
+assets/archi_delay.png filter=lfs diff=lfs merge=lfs -text
+assets/archi_local.png filter=lfs diff=lfs merge=lfs -text
+assets/audio/reference_02_s1.wav filter=lfs diff=lfs merge=lfs -text
+assets/audio/reference_02_s2.wav filter=lfs diff=lfs merge=lfs -text
+assets/audio/reference_en_1.mp3 filter=lfs diff=lfs merge=lfs -text
+assets/audio/reference_en_2.mp3 filter=lfs diff=lfs merge=lfs -text
+assets/audio/reference_en_3.mp3 filter=lfs diff=lfs merge=lfs -text
+assets/audio/reference_zh.wav filter=lfs diff=lfs merge=lfs -text
+assets/audio/reference_zh_0.wav filter=lfs diff=lfs merge=lfs -text
+assets/audio/reference_zh_1.wav filter=lfs diff=lfs merge=lfs -text
+assets/audio/reference_zh_2.wav filter=lfs diff=lfs merge=lfs -text
+assets/audio/reference_zh_3.mp3 filter=lfs diff=lfs merge=lfs -text
+assets/evaluation_moss_audio_tokenizer.png filter=lfs diff=lfs merge=lfs -text
+assets/moss_tts_family.jpeg filter=lfs diff=lfs merge=lfs -text
+assets/moss_tts_realtime.jpeg filter=lfs diff=lfs merge=lfs -text
+assets/moss_voice_generator_winrate.png filter=lfs diff=lfs merge=lfs -text
+moss_tts_realtime/audio/prompt_audio.mp3 filter=lfs diff=lfs merge=lfs -text
+moss_tts_realtime/audio/prompt_audio1.mp3 filter=lfs diff=lfs merge=lfs -text
+moss_tts_realtime/audio/user1.wav filter=lfs diff=lfs merge=lfs -text
+moss_tts_realtime/audio/user2.wav filter=lfs diff=lfs merge=lfs -text
+output_nf4/0_0.wav filter=lfs diff=lfs merge=lfs -text

.gitignore

@@ -0,0 +1,214 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[codz]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py.cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# UV
+#   Similar to Pipfile.lock, it is generally recommended to include uv.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#uv.lock
+
+# poetry
+#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+#poetry.lock
+#poetry.toml
+
+# pdm
+#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
+#   pdm recommends including project-wide configuration in pdm.toml, but excluding .pdm-python.
+#   https://pdm-project.org/en/latest/usage/project/#working-with-version-control
+#pdm.lock
+#pdm.toml
+.pdm-python
+.pdm-build/
+
+# pixi
+#   Similar to Pipfile.lock, it is generally recommended to include pixi.lock in version control.
+#pixi.lock
+#   Pixi creates a virtual environment in the .pixi directory, just like venv module creates one
+#   in the .venv directory. It is recommended not to include this directory in version control.
+.pixi
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.envrc
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+.sii
+.cursor
+.sisyphus
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# PyCharm
+#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
+#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+#  and can be added to the global gitignore or merged into this file.  For a more nuclear
+#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
+#.idea/
+
+# Abstra
+# Abstra is an AI-powered process automation framework.
+# Ignore directories containing user credentials, local state, and settings.
+# Learn more at https://abstra.io/docs
+.abstra/
+
+# Visual Studio Code
+#  Visual Studio Code specific template is maintained in a separate VisualStudioCode.gitignore 
+#  that can be found at https://github.com/github/gitignore/blob/main/Global/VisualStudioCode.gitignore
+#  and can be added to the global gitignore or merged into this file. However, if you prefer, 
+#  you could uncomment the following to ignore the entire vscode folder
+# .vscode/
+
+# Ruff stuff:
+.ruff_cache/
+
+# PyPI configuration file
+.pypirc
+
+# Cursor
+#  Cursor is an AI-powered code editor. `.cursorignore` specifies files/directories to
+#  exclude from AI features like autocomplete and code analysis. Recommended for sensitive data
+#  refer to https://docs.cursor.com/context/ignore-files
+.cursorignore
+.cursorindexingignore
+
+# Marimo
+marimo/_static/
+marimo/_lsp/
+__marimo__/
+dev/*
+
+# Weights
+weights

.gitmodules

@@ -0,0 +1,3 @@
+[submodule "moss_audio_tokenizer"]
+	path = moss_audio_tokenizer
+	url = https://github.com/OpenMOSS/MOSS-Audio-Tokenizer

AMPERE_OPTIMIZATION_PLAN.md

@@ -0,0 +1,181 @@
+# Ampere Optimization Plan for MOSS-TTS Realtime
+
+## Current baseline
+
+- Current service runs on an RTX 3060 (12 GB) and holds about 8.5 GB of VRAM at idle with the model loaded.
+- The backbone already uses SDPA plus `torch.compile(mode="default", dynamic=True)`.
+- The local transformer already uses `StaticCache` plus `torch.compile(..., fullgraph=True)`.
+- The OpenAI-compatible API is single-request by default via `MOSS_TTS_MAX_CONCURRENT=1`.
+- The realtime decode bridge still supports `batch_size=1`, so this is not a batched serving system yet.
+- Multiple RTX 3090 GPUs are currently idle, which makes a GPU move the highest-confidence near-term optimization.
+
+## Goals
+
+- Reduce time-to-first-byte (TTFB) and real-time factor (RTF) on Ampere.
+- Increase throughput without breaking streaming stability or audio quality.
+- Spend engineering time in the order of highest confidence and lowest risk.
+
+## Ranked plan
+
+### 1. Move the service to an RTX 3090 and re-benchmark
+
+Why this comes first:
+
+- It is the easiest change.
+- It has the highest confidence of producing a real improvement.
+- It adds memory headroom for concurrency experiments without changing the model path.
+
+Actions:
+
+- Launch the service on one idle RTX 3090 by overriding `MOSS_TTS_GPU` in `start_moss_tts.sh` or at launch time.
+- Keep the current SDPA plus compile configuration unchanged for the first comparison.
+- Re-run the same short, medium, and long prompt benchmark set used in the optimization report.
+
+Success criteria:
+
+- Record TTFB, RTF, startup time, peak VRAM, and steady-state VRAM on the 3090.
+- Keep this move if it gives a measurable latency win or enables higher safe concurrency.
+
+### 2. Profile where time is actually going
+
+Why this comes second:
+
+- The next bottleneck might be the backbone, the local transformer, codec decode, or Python-side orchestration.
+- Future work should be driven by timing data, not guesses.
+
+Actions:
+
+- Add timing or profiler markers around:
+  - model prefill
+  - backbone decode step loop
+  - local transformer generation
+  - codec decode
+  - audio packaging and response streaming
+- Capture timing breakdowns for at least three prompt sizes on both the 3060 and one 3090.
+
+Success criteria:
+
+- Produce a table showing percentage of end-to-end latency spent in each stage.
+- Use that table to decide whether the next optimization target is the backbone or the codec.
+
+### 3. Increase throughput with controlled concurrency on the 3090
+
+Why this is next:
+
+- The code already gates requests with a semaphore.
+- The 3090 has enough memory headroom to test more than one request in flight.
+- This is a simpler win than trying to force true batching into the realtime decode path.
+
+Actions:
+
+- Keep decode `batch_size=1`.
+- Experiment with `MOSS_TTS_MAX_CONCURRENT=2`, then `3`, while monitoring latency and memory.
+- Measure p50 and p95 TTFB and end-to-end latency under concurrent load.
+
+Success criteria:
+
+- Increase completed requests per minute without unacceptable p95 latency growth.
+- Do not keep a higher concurrency setting if it causes GPU memory pressure, queue thrash, or unstable streaming.
+
+### 4. Wire the codec to ONNX or TensorRT only if profiling shows it matters
+
+Why this is conditional:
+
+- The repo already contains ONNX and TensorRT codec backends.
+- The realtime path does not appear to use them today.
+- This is worthwhile only if codec decode is a meaningful share of total latency.
+
+Actions:
+
+- If profiling shows codec decode is a major bottleneck, add backend selection to the realtime serving path.
+- Export or build the codec backend for the target Ampere GPU.
+- Compare PyTorch, ONNX, and TensorRT codec latency and audio parity.
+
+Success criteria:
+
+- Keep the integration only if it provides a meaningful end-to-end win or raises safe concurrency.
+- Validate that streaming chunk boundaries and audio quality remain acceptable.
+
+### 5. Treat CUDA graphs as a selective experiment, not a blanket recommendation
+
+Why this is not earlier:
+
+- `torch.compile(mode="reduce-overhead")` relies on CUDA graphs.
+- CUDA graphs prefer stable shapes and stable memory addresses.
+- The realtime backbone has growing KV cache and variable request lengths, so a global switch is not automatically a win.
+
+Actions:
+
+- If needed, add shape bucketing for common request sizes before trying broader CUDA-graph capture.
+- Test `reduce-overhead` only on stable-shape regions or after bucketing proves shape churn is limited.
+- Record recompilation behavior, graph re-recording frequency, and memory overhead before adopting it.
+
+Success criteria:
+
+- Keep it only if it improves real end-to-end latency under realistic traffic, not just microbenchmarks.
+- Reject it if dynamic-shape behavior causes graph churn, memory bloat, or operational fragility.
+
+### 6. Treat quantization as a VRAM and throughput project, not an assumed speed win
+
+Why this is later:
+
+- The current repo does not appear to have backbone quantized serving wired into the realtime path.
+- Quantization can help memory a lot, but speedups for autoregressive decode depend heavily on kernels and runtime support.
+- It also adds model-quality risk.
+
+Actions:
+
+- Consider INT8 or INT4 only after phases 1 through 5 are measured.
+- Frame the objective as one of:
+  - fitting more concurrent requests on one GPU
+  - reducing VRAM enough to colocate more services
+  - improving throughput if the chosen backend actually accelerates decode
+- Add listening tests and latency benchmarks before adopting any quantized path.
+
+Success criteria:
+
+- Keep it only if audio quality remains acceptable and measured throughput improves in practice.
+
+### 7. Defer high-risk research work until the simpler wins are exhausted
+
+This includes:
+
+- speculative decoding
+- overlapping codec work on separate CUDA streams
+- major architecture changes to the streaming pipeline
+
+Why this is last:
+
+- These ideas may help, but they are not low-risk or quick-return changes.
+- They should only start after the system has a stable benchmark harness and clear bottleneck data.
+
+## Decision gates
+
+At each phase, record:
+
+- p50 TTFB
+- p95 TTFB
+- end-to-end latency
+- RTF
+- peak VRAM
+- steady-state VRAM
+- startup and warmup time
+- requests per minute at a fixed latency target
+- basic audio quality checks
+
+Do not move to the next invasive phase unless the previous phase is measured and documented.
+
+## Immediate next steps
+
+1. Move the service to one idle RTX 3090 and rerun the existing benchmark set.
+2. Add timing breakdowns to the realtime path.
+3. Decide whether the next engineering target is concurrency tuning or codec acceleration based on those numbers.
+
+## Relevant code paths
+
+- `start_moss_tts.sh`
+- `moss_tts_realtime/app.py`
+- `moss_tts_realtime/openai_api.py`
+- `moss_tts_realtime/mossttsrealtime/streaming_mossttsrealtime.py`
+- `moss_audio_tokenizer/onnx/inference.py`
+- `moss_audio_tokenizer/trt/inference.py`

LICENSE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright 2026 OpenMOSS Team, Fudan University, SII and MOSI
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

MANIFEST.in

@@ -0,0 +1,22 @@
+include README.md
+include README_zh.md
+include LICENSE
+include pyproject.toml
+
+graft assets
+graft docs
+graft moss_tts_delay
+graft moss_tts_local
+graft moss_tts_realtime
+graft moss_audio_tokenizer
+
+prune .git
+prune .github
+prune .vscode
+prune moss_tts.egg-info
+
+global-exclude __pycache__
+global-exclude __pycache__/*
+global-exclude *.py[cod]
+global-exclude .DS_Store
+global-exclude *.so

OPTIMIZATION_REPORT.md

@@ -0,0 +1,296 @@
+# MOSS-TTS Realtime — RTX 3090 Performance Optimization Report
+
+**Date**: 2026-03-13
+**GPU**: NVIDIA GeForce RTX 3090 (24 GB, Ampere SM 8.6)
+**Stack**: PyTorch 2.9.1+cu128, transformers 5.0.0, flash-attn 2.8.3, triton 3.5.1
+
+---
+
+## Executive Summary
+
+Optimized the MOSS-TTS-Realtime OpenAI-compatible server for maximum throughput
+and minimum latency on the RTX 3090. Through four targeted changes to the
+inference pipeline, achieved:
+
+| Metric     | Before  | After   | Improvement |
+|------------|---------|---------|-------------|
+| **RTF**    | 0.8044  | 0.3352  | **−58%** (2.4× throughput) |
+| **TTFB**   | 1557 ms | 586 ms  | **−62%** (2.7× faster first chunk) |
+| **Min RTF**| 0.7858  | 0.3073  | Best-case 3.3× realtime |
+
+40.5 seconds of audio generated in 13.4 seconds (wall-clock) across 7 test
+sentences of varying length (12–202 characters).
+
+---
+
+## Architecture Overview
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│  OpenAI API  (FastAPI + uvicorn)  :8012                     │
+│    POST /v1/audio/speech                                    │
+├─────────────────────────────────────────────────────────────┤
+│  Streaming Pipeline                                         │
+│    TokenChunkStream → StreamingSession → AudioStreamDecoder │
+├─────────────────────────────────────────────────────────────┤
+│  Backbone Transformer  (Qwen3-1.7B, 28 layers)             │
+│    2048 hidden, 16 heads, 8 KV heads, bfloat16             │
+│    → Produces hidden states per token step                  │
+├─────────────────────────────────────────────────────────────┤
+│  Local Transformer  (4 layers, 2048 hidden)                 │
+│    16 sequential RVQ codebook steps per backbone step       │
+│    → Produces 16-channel audio tokens                       │
+├─────────────────────────────────────────────────────────────┤
+│  MOSS Audio Tokenizer (codec)                               │
+│    Decodes RVQ tokens → 24 kHz waveform                     │
+│    Streaming mode with crossfade overlap                    │
+└─────────────────────────────────────────────────────────────┘
+```
+
+---
+
+## Bottlenecks Identified & Fixes Applied
+
+### 1. Local Transformer: DynamicCache + No Compilation (HIGH IMPACT)
+
+**Problem**: When the backbone used `flash_attention_2`, the local transformer
+inherited the same attention implementation. This forced `DynamicCache` and
+**disabled `torch.compile`** for the 16-step RVQ decode loop — the innermost
+hot loop of the entire pipeline.
+
+The local transformer processes sequences of only 16 tokens (one per RVQ
+codebook). Flash attention provides negligible benefit at this length, while
+`StaticCache` + `torch.compile(fullgraph=True)` can fuse the entire loop into
+optimized Triton kernels.
+
+**Fix** (`streaming_mossttsrealtime.py`, `inferencer.py`):
+- Override local transformer config `_attn_implementation` → `"sdpa"`
+- Force `StaticCache(max_cache_len=16)` regardless of backbone attention
+- Enable `torch.compile(fullgraph=True)` for the local transformer always
+
+```python
+# Before: cache/compile depended on backbone attn
+self._use_dynamic_local_cache = attn_impl == "flash_attention_2"  # True → no compile
+self._should_compile_local_transformer = not self._use_dynamic_local_cache  # False
+
+# After: always StaticCache + compile for local transformer
+local_cfg._attn_implementation = "sdpa"
+self._use_dynamic_local_cache = False
+self._should_compile_local_transformer = True
+```
+
+### 2. Backbone: SDPA + torch.compile (HIGH IMPACT)
+
+**Problem**: The backbone Qwen3 language model ran uncompiled. Additionally,
+`flash_attention_2` prevented `torch.compile` from generating fused Triton
+kernels for the attention + MLP blocks.
+
+**Finding**: Benchmarking revealed that on the RTX 3090 (Ampere),
+`SDPA + torch.compile(dynamic=True)` dramatically outperforms
+`flash_attention_2` for typical TTS sequence lengths:
+
+| Config                        | RTF   | TTFB    |
+|-------------------------------|-------|---------|
+| flash_attn2, no compile       | 0.805 | 1557 ms |
+| flash_attn2 + compile         | 0.490 | 950 ms  |
+| **SDPA + compile (dynamic)**  | **0.335** | **586 ms** |
+
+**Fix** (`app.py`):
+- Switch default attention to `"sdpa"` (auto-detected)
+- Add `torch.compile(mode="default", dynamic=True)` for the backbone
+- `dynamic=True` prevents shape-triggered recompilation as KV cache grows
+
+```python
+model.language_model = torch.compile(
+    model.language_model,
+    mode="default",
+    fullgraph=False,
+    dynamic=True,
+)
+```
+
+### 3. No Startup Warmup (HIGH TTFB IMPACT)
+
+**Problem**: `WARMUP_ON_START` defaulted to `false`. The first user request
+suffered a 30–70 second cold-start penalty while `torch.compile` generated
+and cached Triton kernels.
+
+**Fix** (`openai_api.py`):
+- Default `WARMUP_ON_START` to `true`
+- Run 2 warmup generations with different text lengths during `lifespan()`
+- Server starts accepting traffic only after caches are hot (~2.5 min startup)
+
+### 4. CUDA Runtime Enhancements (LOW-MEDIUM IMPACT)
+
+**Fix** (`app.py`):
+- Increased `torch._dynamo.config.cache_size_limit` from 64 → 128
+- Added `torch._dynamo.config.suppress_errors = True` for graceful fallback
+- Added CUDA memory pool pre-allocation (`set_per_process_memory_fraction`)
+
+---
+
+## Why SDPA Beats Flash Attention on 3090
+
+Flash Attention 2 is a hand-written CUDA kernel optimized for long sequences
+(>512 tokens). It bypasses PyTorch's operator fusion.
+
+SDPA (`torch.nn.functional.scaled_dot_product_attention`) is a PyTorch-native
+op that `torch.compile` can fuse with surrounding operations (layernorm, MLP,
+residual connections) into a single Triton kernel graph.
+
+For MOSS-TTS-Realtime:
+- **Backbone decode step**: 1 token input, KV cache grows to ~100–400 tokens
+- **Local transformer**: 16 tokens total
+
+At these lengths, the kernel launch overhead of flash_attn's separate CUDA
+kernel dominates over its memory-access advantage. `torch.compile` + SDPA
+generates fused Triton kernels that eliminate this overhead entirely.
+
+---
+
+## Benchmark Details
+
+### Test Configuration
+- 7 sentences, 12–202 characters
+- Voice preset: alloy (prompt audio cloning)
+- Response format: WAV (no codec encoding overhead)
+- All requests sequential (no concurrency)
+
+### Before (Baseline)
+```
+flash_attention_2, no torch.compile, no warmup
+  [1] RTF=0.8213 TTFB=1579ms audio=5.56s chars=79
+  [2] RTF=0.7858 TTFB=1539ms audio=6.35s chars=97
+  [3] RTF=0.8061 TTFB=1553ms audio=7.41s chars=113
+  Avg: RTF=0.8044  TTFB=1557ms
+```
+
+### After (Optimized)
+```
+SDPA + torch.compile(dynamic=True), StaticCache local transformer, warmup
+  [1] RTF=0.3578 TTFB=455ms audio=1.28s chars=12
+  [2] RTF=0.3182 TTFB=623ms audio=2.71s chars=32
+  [3] RTF=0.3073 TTFB=596ms audio=7.28s chars=79
+  [4] RTF=0.4287 TTFB=590ms audio=6.15s chars=97
+  [5] RTF=0.3091 TTFB=623ms audio=6.15s chars=113
+  [6] RTF=0.3128 TTFB=602ms audio=7.55s chars=144
+  [7] RTF=0.3126 TTFB=616ms audio=9.40s chars=202
+  Avg: RTF=0.3352  TTFB=586ms
+```
+
+---
+
+## Files Modified
+
+| File | Changes |
+|------|---------|
+| `moss_tts_realtime/mossttsrealtime/streaming_mossttsrealtime.py` | Force local transformer → SDPA + StaticCache + torch.compile |
+| `moss_tts_realtime/inferencer.py` | Same local transformer fix (standalone inference path) |
+| `moss_tts_realtime/app.py` | Backbone torch.compile, CUDA runtime enhancements, dynamo config |
+| `moss_tts_realtime/openai_api.py` | Default SDPA, warmup on start, multi-shape warmup |
+
+---
+
+## Server Configuration
+
+The optimized server runs at `http://127.0.0.1:8012` with:
+- `MOSS_TTS_ATTN_IMPLEMENTATION=auto` (resolves to `sdpa`)
+- `MOSS_TTS_WARMUP_ON_START=true` (default)
+- `MOSS_TTS_COMPILE_BACKBONE=true` (default)
+- Startup time: ~2.5 minutes (model load + warmup compilation)
+
+To force flash_attention_2 (if needed for other hardware):
+```bash
+export MOSS_TTS_ATTN_IMPLEMENTATION=flash_attention_2
+```
+
+To disable backbone compilation (if stability issues arise):
+```bash
+export MOSS_TTS_COMPILE_BACKBONE=false
+```
+
+---
+
+## Remaining Optimization Opportunities
+
+1. **ONNX/TensorRT codec decode**: The audio tokenizer codec runs as PyTorch.
+   ONNX or TRT backends exist in `moss_audio_tokenizer/` but aren't wired
+   into the streaming path. Could reduce decode latency.
+
+2. **CUDA graphs for fixed-shape decode**: If inputs were padded to discrete
+   bucket sizes, `reduce-overhead` mode could capture full CUDA graphs for
+   near-zero kernel launch overhead.
+
+3. **Quantized backbone**: The backbone runs in bfloat16. INT8/INT4
+   quantization (via GPTQ, AWQ, or torch.ao) could further reduce compute.
+
+4. **Speculative decoding**: Pre-generate multiple RVQ frames speculatively
+   and verify — could reduce effective RTF further.
+
+5. **Multi-stream codec decode**: Overlap codec decoding with backbone
+   generation using separate CUDA streams.
+
+---
+
+## RTX 3060 Deployment (2026-03-13 Update)
+
+### Goal
+Run the optimized server on an RTX 3060 (12 GB VRAM) alongside existing 3090 workloads.
+
+### VRAM Challenge & Fix
+
+| Component | float32 | bfloat16 | float16 |
+|-----------|---------|----------|---------|
+| MOSS-TTS-Realtime backbone | — | 4,400 MiB | — |
+| MOSS-Audio-Tokenizer codec | 6,860 MiB | ❌ unsupported¹ | 3,430 MiB |
+| **Total on 3060** | > 12 GB ❌ | — | **~9,200 MiB ✅** |
+
+¹ The codec's custom CUDA kernels raise "Got unsupported ScalarType BFloat16" — float16 is the only viable half-precision dtype.
+
+**Fix**: `_load_codec()` in `app.py` now explicitly loads the codec with `torch_dtype=torch.float16` and wraps it in `_BF16CodecWrapper` (autocast float16 context). The backbone remains in bfloat16.
+
+### Results on RTX 3060
+
+- **VRAM used**: 9,174 MiB / 12,288 MiB  
+- **Server startup**: ~90 seconds (model load + torch.compile + 2× warmup)
+- **Inference latency** (warm, SDPA + torch.compile):
+  - Short sentence (20 chars): ~1.8 s total
+  - Medium sentence (87 chars): ~3.0–4.7 s total
+- **Health check**: `GET /health` → `{"status":"ok","device":"cuda:0","attn_implementation":"sdpa"}`
+
+### Deployment Infrastructure
+
+**Shell launcher** (`start_moss_tts.sh` — default launch method):
+```bash
+./start_moss_tts.sh
+# or with overrides:
+MOSS_TTS_GPU=GPU-other MOSS_TTS_PORT=8013 ./start_moss_tts.sh
+```
+
+**Systemd service** (auto-start on boot):
+```bash
+sudo systemctl enable --now moss-tts.service
+sudo journalctl -u moss-tts -f
+```
+
+Service configuration (`/etc/systemd/system/moss-tts.service`):
+- GPU: `CUDA_VISIBLE_DEVICES=GPU-cbfc8a5f-0df1-ca71-f704-0d09a707d2ac` (RTX 3060)
+- All optimizations enabled: `MOSS_TTS_COMPILE_BACKBONE=true`, `MOSS_TTS_WARMUP_ON_START=true`
+- Restart on failure with 10s backoff, 300s startup timeout
+
+**API endpoint** (OpenAI-compatible):
+```
+POST http://0.0.0.0:8012/v1/audio/speech
+GET  http://0.0.0.0:8012/v1/audio/models
+GET  http://0.0.0.0:8012/v1/audio/voices
+GET  http://0.0.0.0:8012/health
+```
+
+### Git History
+
+| Commit | Description |
+|--------|-------------|
+| `6a89aff` | feat: RTX 3090 performance optimizations (flash_attn, torch.compile, SDPA, warmup) |
+| `2f0b604` | fix: use float16 for codec to support RTX 3060, add launcher script |
+
+Repository: https://github.com/groxaxo/MOSS-TTS

README.md

@@ -0,0 +1,548 @@
+# MOSS-TTS Family
+
+<br>
+
+<p align="center">
+  <img src="./assets/OpenMOSS_Logo.png" height="70" align="middle" />
+  &nbsp;&nbsp;&nbsp;&nbsp;
+  <img src="./assets/mosi-logo.png" height="50" align="middle" />
+</p>
+
+
+
+
+<div align="center">
+  <a href="https://clawhub.ai/luogao2333/moss-tts-voice"><img src="https://img.shields.io/badge/🦞_OpenClaw-Skills-8A2BE2" alt="OpenClaw"></a>
+  <a href="https://huggingface.co/collections/OpenMOSS-Team/moss-tts"><img src="https://img.shields.io/badge/Huggingface-Models-orange?logo=huggingface&amp"></a>
+  <a href="https://modelscope.cn/collections/OpenMOSS-Team/MOSS-TTS"><img src="https://img.shields.io/badge/ModelScope-Models-lightgrey?logo=modelscope&amp"></a>
+  <a href="https://mosi.cn/#models"><img src="https://img.shields.io/badge/Blog-View-blue?logo=internet-explorer&amp"></a>
+  <a href="https://github.com/OpenMOSS/MOSS-TTS"><img src="https://img.shields.io/badge/Arxiv-Coming%20soon-red?logo=arxiv&amp"></a>
+
+  <a href="https://studio.mosi.cn"><img src="https://img.shields.io/badge/AIStudio-Try-green?logo=internet-explorer&amp"></a>
+  <a href="https://studio.mosi.cn/docs/moss-tts"><img src="https://img.shields.io/badge/API-Docs-00A3FF?logo=fastapi&amp"></a>
+  <a href="https://x.com/Open_MOSS"><img src="https://img.shields.io/badge/Twitter-Follow-black?logo=x&amp"></a>
+  <a href="https://discord.gg/fvm5TaWjU3"><img src="https://img.shields.io/badge/Discord-Join-5865F2?logo=discord&amp"></a>
+  <a href="./assets/wechat.jpg"><img src="https://img.shields.io/badge/WeChat-Join-07C160?logo=wechat&amp;logoColor=white" alt="WeChat"></a>
+</div>
+
+
+[English](README.md) | [简体中文](README_zh.md)
+
+
+MOSS‑TTS Family is an open‑source **speech and sound generation model family** from [MOSI.AI](https://mosi.cn/#hero) and the [OpenMOSS team](https://www.open-moss.com/). It is designed for **high‑fidelity**, **high‑expressiveness**, and **complex real‑world scenarios**, covering stable long‑form speech, multi‑speaker dialogue, voice/character design, environmental sound effects, and real‑time streaming TTS.
+
+## News
+* 2026.3.10: ⚡️ Significantly optimized the VRAM usage of llama.cpp inference pipeline. Now 8B model fits onto 8GB GPUs !
+* 2026.3.4: 🚀 Added **PyTorch-free inference support** — enabling lightweight on-device deployment via **llama.cpp + ONNX Runtime**. Quantized **GGUF weights** are released at [OpenMOSS-Team/MOSS-TTS-GGUF](https://huggingface.co/OpenMOSS-Team/MOSS-TTS-GGUF), and the **ONNX audio tokenizer** is available at [OpenMOSS-Team/MOSS-Audio-Tokenizer-ONNX](https://huggingface.co/OpenMOSS-Team/MOSS-Audio-Tokenizer-ONNX). See the [llama.cpp backend](#llamacpp-backend-torch-free-inference) for details.
+* 2026.3.4: 🎉 We add MOSS-TTS skills in [ClawHub](https://clawhub.ai) of 🦞 OpenClaw: [feishu-voice-tts](https://clawhub.ai/helloeveryworlds/feishu-voice-tts) and [moss-tts-voice](https://clawhub.ai/luogao2333/moss-tts-voice).
+* 2026.2.10: 🎉🎉🎉 We have released [MOSS-TTS Family](https://huggingface.co/collections/OpenMOSS-Team/moss-tts). Check our [Blog](https://mosi.cn/#models) for more details! Our **Huggingface Space** is here: [MOSS-TTS](https://huggingface.co/spaces/OpenMOSS-Team/MOSS-TTS), [MOSS-TTSD-v1.0](https://huggingface.co/spaces/OpenMOSS-Team/MOSS-TTSD-v1.0), [MOSS-VoiceGenerator](https://huggingface.co/spaces/OpenMOSS-Team/MOSS-VoiceGenerator).
+
+
+## Demo
+
+<div align="center">
+  <video src="https://gist.github.com/user-attachments/assets/fdce9f66-20ec-45e8-9615-89606ae2fbe8" width="70%" poster=""> </video>
+</div>
+
+## Contents
+
+- [Introduction](#introduction)
+- [Model Architecture](#model-architecture)
+- [Released Models](#released-models)
+- [Supported Languages](#supported-languages)
+- [Quickstart](#quickstart)
+  - [OpenClaw API Skills](#openclaw-api-skills)
+  - [Environment Setup](#environment-setup)
+  - [(Optional) Install FlashAttention 2](#optional-install-flashattention-2)
+  - [MOSS-TTS Basic Usage](#moss-tts-basic-usage)
+- [Fine-Tuning](#fine-tuning)
+- [llama.cpp Backend (Torch-Free Inference)](#llamacpp-backend-torch-free-inference)
+- [Evaluation](#evaluation)
+  - [MOSS-TTS](#moss-tts-seed-tts-eval)
+  - [MOSS-TTSD](#moss-ttsd-subjective--ttsd-eval)
+  - [MOSS-VoiceGenerator](#moss-voicegenerator-subjective)
+- [MOSS-Audio-Tokenizer](#moss-audio-tokenizer)
+  - [Introduction](#mat-intro)
+  - [Model Weights](#model-weights)
+  - [Objective Reconstruction Evaluation](#objective-reconstruction-evaluation)
+
+
+## Introduction
+
+<p align="center">
+  <img src="./assets/moss_tts_family.jpeg" width="85%" />
+</p>
+
+When a single piece of audio needs to **sound like a real person**, **pronounce every word accurately**, **switch speaking styles across content**, **remain stable over tens of minutes**, and **support dialogue, role‑play, and real‑time interaction**, a single TTS model is often not enough. The **MOSS‑TTS Family** breaks the workflow into five production‑ready models that can be used independently or composed into a complete pipeline.
+
+- **MOSS‑TTS**: The flagship production model featuring high fidelity and optimal zero-shot voice cloning. It supports **long-speech generation**, **fine-grained control over Pinyin, phonemes, and duration**, as well as **multilingual/code-switched synthesis**.
+- **MOSS‑TTSD**: A spoken dialogue generation model for expressive, multi-speaker, and ultra-long dialogues. The new **v1.0 version** achieves **industry-leading performance on objective metrics** and **outperformed top closed-source models like Doubao and Gemini 2.5-pro** in subjective evaluations. You can visit the [MOSS-TTSD repository](https://github.com/OpenMOSS/MOSS-TTSD) for details.
+- **MOSS‑VoiceGenerator**: An open-source voice design model capable of generating diverse voices and styles directly from text prompts, **without any reference speech**. It unifies voice design, style control, and synthesis, functioning independently or as a design layer for downstream TTS. Its performance **surpasses other top-tier voice design models in arena ratings**.
+- **MOSS‑TTS‑Realtime**: A multi-turn context-aware model for real-time voice agents. It uses incremental synthesis to ensure natural and coherent replies, making it **ideal for building low-latency voice agents when paired with text models**.
+- **MOSS‑SoundEffect**: A content creation model specialized in **sound effect generation** with wide category coverage and controllable duration. It generates audio for natural environments, urban scenes, biological sounds, human actions, and musical fragments, suitable for film, games, and interactive experiences.
+
+
+## Model Architecture
+
+We train **MossTTSDelay** and **MossTTSLocal** as complementary baselines under one training/evaluation setup: **Delay** emphasizes long-context stability, inference speed, and production readiness, while **Local** emphasizes lightweight flexibility and strong objective performance for streaming-oriented systems. Together they provide reproducible references for deployment and research.
+
+**MossTTSRealtime** is not a third comparison baseline; it is a capability-driven design for voice agents. By modeling multi-turn context from both prior text and user acoustics, it delivers low-latency streaming speech that stays coherent and voice-consistent across turns.
+
+
+| Architecture  | Core Mechanism | Arch Details |
+|---|---|---|
+| `MossTTSDelay` |  Multi‑head parallel RVQ prediction with delay‑pattern scheduling | [![Arch Details](https://img.shields.io/badge/Model%20Card-View-blue?logo=markdown)](moss_tts_delay/README.md) |
+| `MossTTSLocal` | Time‑synchronous RVQ blocks with a depth transformer | [![Arch Details](https://img.shields.io/badge/Model%20Card-View-blue?logo=markdown)](moss_tts_local/README.md) |
+| `MossTTSRealtime` | Hierarchical text–audio inputs for realtime synthesis | [![Arch Details](https://img.shields.io/badge/Model%20Card-View-blue?logo=markdown)](moss_tts_realtime/README.md) |
+
+## Released Models
+
+
+| Model | Architecture | Size | Model Card | Hugging Face | ModelScope |
+|---|---|---:|---|---|---|
+| **MOSS-TTS** | `MossTTSDelay` | 8B | [![Model Card](https://img.shields.io/badge/Model%20Card-View-blue?logo=markdown)](docs/moss_tts_model_card.md) | [![Hugging Face](https://img.shields.io/badge/Huggingface-Model-orange?logo=huggingface)](https://huggingface.co/OpenMOSS-Team/MOSS-TTS) | [![ModelScope](https://img.shields.io/badge/ModelScope-Model-lightgrey?logo=modelscope)](https://modelscope.cn/models/openmoss/MOSS-TTS) |
+|  | `MossTTSLocal` | 1.7B | [![Model Card](https://img.shields.io/badge/Model%20Card-View-blue?logo=markdown)](docs/moss_tts_model_card.md) | [![Hugging Face](https://img.shields.io/badge/Huggingface-Model-orange?logo=huggingface)](https://huggingface.co/OpenMOSS-Team/MOSS-TTS-Local-Transformer) | [![ModelScope](https://img.shields.io/badge/ModelScope-Model-lightgrey?logo=modelscope)](https://modelscope.cn/models/openmoss/MOSS-TTS-Local-Transformer) |
+| **MOSS‑TTSD‑V1.0** | `MossTTSDelay` | 8B | [![Model Card](https://img.shields.io/badge/Model%20Card-View-blue?logo=markdown)](docs/moss_ttsd_model_card.md) | [![Hugging Face](https://img.shields.io/badge/Huggingface-Model-orange?logo=huggingface)](https://huggingface.co/OpenMOSS-Team/MOSS-TTSD-v1.0) | [![ModelScope](https://img.shields.io/badge/ModelScope-Model-lightgrey?logo=modelscope)](https://modelscope.cn/models/openmoss/MOSS-TTSD-v1.0) |
+| **MOSS‑VoiceGenerator** | `MossTTSDelay` | 1.7B | [![Model Card](https://img.shields.io/badge/Model%20Card-View-blue?logo=markdown)](docs/moss_voice_generator_model_card.md) | [![Hugging Face](https://img.shields.io/badge/Huggingface-Model-orange?logo=huggingface)](https://huggingface.co/OpenMOSS-Team/MOSS-VoiceGenerator) | [![ModelScope](https://img.shields.io/badge/ModelScope-Model-lightgrey?logo=modelscope)](https://modelscope.cn/models/openmoss/MOSS-VoiceGenerator) |
+| **MOSS‑SoundEffect** | `MossTTSDelay` | 8B | [![Model Card](https://img.shields.io/badge/Model%20Card-View-blue?logo=markdown)](docs/moss_sound_effect_model_card.md) | [![Hugging Face](https://img.shields.io/badge/Huggingface-Model-orange?logo=huggingface)](https://huggingface.co/OpenMOSS-Team/MOSS-SoundEffect) | [![ModelScope](https://img.shields.io/badge/ModelScope-Model-lightgrey?logo=modelscope)](https://modelscope.cn/models/openmoss/MOSS-SoundEffect) |
+| **MOSS‑TTS‑Realtime** | `MossTTSRealtime` | 1.7B | [![Model Card](https://img.shields.io/badge/Model%20Card-View-blue?logo=markdown)](docs/moss_tts_realtime_model_card.md) | [![Hugging Face](https://img.shields.io/badge/Huggingface-Model-orange?logo=huggingface)](https://huggingface.co/OpenMOSS-Team/MOSS-TTS-Realtime) | [![ModelScope](https://img.shields.io/badge/ModelScope-Model-lightgrey?logo=modelscope)](https://modelscope.cn/models/openmoss/MOSS-TTS-Realtime) |
+
+## Supported Languages
+
+MOSS-TTS, MOSS-TTSD and MOSS-TTS-Realtime currently supports **20 languages**:
+
+| Language | Code | Flag | Language | Code | Flag | Language | Code | Flag |
+|---|---|---|---|---|---|---|---|---|
+| Chinese | zh | 🇨🇳 | English | en | 🇺🇸 | German | de | 🇩🇪 |
+| Spanish | es | 🇪🇸 | French | fr | 🇫🇷 | Japanese | ja | 🇯🇵 |
+| Italian | it | 🇮🇹 | Hungarian | hu | 🇭🇺 | Korean | ko | 🇰🇷 |
+| Russian | ru | 🇷🇺 | Persian (Farsi) | fa | 🇮🇷 | Arabic | ar | 🇸🇦 |
+| Polish | pl | 🇵🇱 | Portuguese | pt | 🇵🇹 | Czech | cs | 🇨🇿 |
+| Danish | da | 🇩🇰 | Swedish | sv | 🇸🇪 | | | |
+| Greek | el | 🇬🇷 | Turkish | tr | 🇹🇷 |  |  |  |
+
+
+## Quickstart
+
+### OpenClaw API Skills
+
+We add MOSS-TTS skills in [ClawHub](https://clawhub.ai) of 🦞 OpenClaw. You can get your API key from [MOSI AI Studio](https://studio.mosi.cn).
+
+| Skill | Description | Install |
+|---|---|---|
+| [`feishu-voice-tts`](https://clawhub.ai/helloeveryworlds/feishu-voice-tts) | Send voice messages in Feishu | `clawhub install feishu-voice-tts` |
+| [`moss-tts-voice`](https://clawhub.ai/luogao2333/moss-tts-voice) | Call MOSS-TTS API to generate speech | `clawhub install moss-tts-voice` |
+
+### Environment Setup
+
+We recommend a clean, isolated Python environment with **Transformers 5.0.0** to avoid dependency conflicts.
+
+#### Using Conda
+
+```bash
+conda create -n moss-tts python=3.12 -y
+conda activate moss-tts
+```
+
+Install all required dependencies:
+
+```bash
+git clone https://github.com/OpenMOSS/MOSS-TTS.git
+cd MOSS-TTS
+pip install --extra-index-url https://download.pytorch.org/whl/cu128 -e ".[torch-runtime]"
+```
+
+#### Using `uv`
+
+```bash
+# Install uv first: https://docs.astral.sh/uv/getting-started/installation/
+git clone https://github.com/OpenMOSS/MOSS-TTS.git
+cd MOSS-TTS
+uv venv --python 3.12 .venv
+source .venv/bin/activate
+uv pip install --torch-backend cu128 -e ".[torch-runtime]"
+```
+
+#### (Optional) Install FlashAttention 2
+
+For better speed and lower GPU memory usage, you can install FlashAttention 2 if your hardware supports it.
+
+If you use Conda/pip:
+
+```bash
+pip install --extra-index-url https://download.pytorch.org/whl/cu128 -e ".[torch-runtime,flash-attn]"
+```
+
+If your machine has limited RAM and many CPU cores, you can cap build parallelism:
+
+```bash
+MAX_JOBS=4 pip install --extra-index-url https://download.pytorch.org/whl/cu128 -e ".[torch-runtime,flash-attn]"
+```
+
+If you use `uv`:
+
+```bash
+uv pip install --torch-backend cu128 -e ".[torch-runtime,flash-attn]"
+```
+
+If your machine has limited RAM and many CPU cores, you can cap build parallelism:
+
+```bash
+MAX_JOBS=4 uv pip install --torch-backend cu128 -e ".[torch-runtime,flash-attn]"
+```
+
+Notes:
+- Dependencies are managed in `pyproject.toml`, which currently pins `torch==2.9.1+cu128` and `torchaudio==2.9.1+cu128`.
+- In `uv`, `--torch-backend cu128` lets uv fetch compatible PyTorch CUDA wheels and resolve the rest from PyPI with the default safe index strategy.
+- If you need another backend, replace `cu128` with your target (for example, `cpu`, `cu126`).
+- If FlashAttention 2 fails to build on your machine, you can skip it and use the default attention backend.
+- FlashAttention 2 is only available on supported GPUs and is typically used with `torch.float16` or `torch.bfloat16`.
+
+#### OpenAI-compatible FastAPI server for Open WebUI
+
+The realtime model now includes a FastAPI server with OpenAI-style TTS endpoints:
+
+```bash
+conda activate moss-tts
+cd MOSS-TTS
+MOSS_TTS_DEVICE=cuda:0 moss-tts-realtime-openai
+```
+
+Available routes:
+
+- `POST /v1/audio/speech`
+- `GET /v1/audio/models`
+- `GET /v1/audio/voices`
+
+Compatibility aliases are also exposed at `/audio/speech`, `/audio/models`, and `/audio/voices`.
+
+For Open WebUI, point the custom TTS OpenAI base URL to either:
+
+- `http://<host>:8012/v1`
+- `http://<host>:8012`
+
+The server reuses the MOSS-TTS-Realtime streaming backend, caches bundled prompt voices, and auto-selects the best available attention backend (`flash_attention_2` on supported Ampere GPUs when `flash-attn` is installed, otherwise `sdpa`).
+
+
+<a id="moss-tts-basic-usage"></a>
+### MOSS‑TTS Basic Usage
+
+If you prefer Gradio demos, we provide 4 scripts for the main models:
+
+| Model | Script | Run |
+|---|---|---|
+| MOSS-TTS | [clis/moss_tts_app.py](clis/moss_tts_app.py) |
+| MOSS-TTSD | [clis/moss_ttsd_app.py](clis/moss_ttsd_app.py) | 
+| MOSS-VoiceGenerator | [clis/moss_voice_generator_app.py](clis/moss_voice_generator_app.py) |
+| MOSS-SoundEffect | [clis/moss_sound_effect_app.py](clis/moss_sound_effect_app.py) | 
+
+For the MOSS-TTS-Realtime Gradio demo, please refer to [MOSS-TTS-Realtime Model Card](docs/moss_tts_realtime_model_card.md)
+
+```python
+from pathlib import Path
+import importlib.util
+import torch
+import torchaudio
+from transformers import AutoModel, AutoProcessor
+# Disable the broken cuDNN SDPA backend
+torch.backends.cuda.enable_cudnn_sdp(False)
+# Keep these enabled as fallbacks
+torch.backends.cuda.enable_flash_sdp(True)
+torch.backends.cuda.enable_mem_efficient_sdp(True)
+torch.backends.cuda.enable_math_sdp(True)
+
+
+pretrained_model_name_or_path = "OpenMOSS-Team/MOSS-TTS"
+device = "cuda" if torch.cuda.is_available() else "cpu"
+dtype = torch.bfloat16 if device == "cuda" else torch.float32
+
+def resolve_attn_implementation() -> str:
+    # Prefer FlashAttention 2 when package + device conditions are met.
+    if (
+        device == "cuda"
+        and importlib.util.find_spec("flash_attn") is not None
+        and dtype in {torch.float16, torch.bfloat16}
+    ):
+        major, _ = torch.cuda.get_device_capability()
+        if major >= 8:
+            return "flash_attention_2"
+
+    # CUDA fallback: use PyTorch SDPA kernels.
+    if device == "cuda":
+        return "sdpa"
+
+    # CPU fallback.
+    return "eager"
+
+
+attn_implementation = resolve_attn_implementation()
+print(f"[INFO] Using attn_implementation={attn_implementation}")
+
+processor = AutoProcessor.from_pretrained(
+    pretrained_model_name_or_path,
+    trust_remote_code=True,
+)
+processor.audio_tokenizer = processor.audio_tokenizer.to(device)
+
+text_1 = "亲爱的你，\n你好呀。\n\n今天，我想用最认真、最温柔的声音，对你说一些重要的话。\n这些话，像一颗小小的星星，希望能在你的心里慢慢发光。\n\n首先，我想祝你——\n每天都能平平安安、快快乐乐。\n\n希望你早上醒来的时候，\n窗外有光，屋子里很安静，\n你的心是轻轻的，没有着急，也没有害怕。\n\n希望你吃饭的时候胃口很好，\n走路的时候脚步稳稳，\n晚上睡觉的时候，能做一个又一个甜甜的梦。\n\n我希望你能一直保持好奇心。\n对世界充满问题，\n对天空、星星、花草、书本和故事感兴趣。\n当你问“为什么”的时候，\n希望总有人愿意认真地听你说话。\n\n我也希望你学会温柔。\n温柔地对待朋友，\n温柔地对待小动物，\n也温柔地对待自己。\n\n如果有一天你犯了错，\n请不要太快责怪自己，\n因为每一个认真成长的人，\n都会在路上慢慢学会更好的方法。\n\n愿你拥有勇气。\n当你站在陌生的地方时，\n当你第一次举手发言时，\n当你遇到困难、感到害怕的时候，\n希望你能轻轻地告诉自己：\n“我可以试一试。”\n\n就算没有一次成功，也没有关系。\n失败不是坏事，\n它只是告诉你，你正在努力。\n\n我希望你学会分享快乐。\n把开心的事情告诉别人，\n把笑声送给身边的人，\n因为快乐被分享的时候，\n会变得更大、更亮。\n\n如果有一天你感到难过，\n我希望你知道——\n难过并不丢脸，\n哭泣也不是软弱。\n\n愿你能找到一个安全的地方，\n慢慢把心里的话说出来，\n然后再一次抬起头，看见希望。\n\n我还希望你能拥有梦想。\n这个梦想也许很大，\n也许很小，\n也许现在还说不清楚。\n\n没关系。\n梦想会和你一起长大，\n在时间里慢慢变得清楚。\n\n最后，我想送你一个最最重要的祝福：\n\n愿你被世界温柔对待，\n也愿你成为一个温柔的人。\n\n愿你的每一天，\n都值得被记住，\n都值得被珍惜。\n\n亲爱的你，\n请记住，\n你是独一无二的，\n你已经很棒了，\n而你的未来，\n一定会慢慢变得闪闪发光。\n\n祝你健康、勇敢、幸福，\n祝你永远带着笑容向前走。"
+text_2 = "We stand on the threshold of the AI era.\nArtificial intelligence is no longer just a concept in laboratories, but is entering every industry, every creative endeavor, and every decision. It has learned to see, hear, speak, and think, and is beginning to become an extension of human capabilities. AI is not about replacing humans, but about amplifying human creativity, making knowledge more equitable, more efficient, and allowing imagination to reach further. A new era, jointly shaped by humans and intelligent systems, has arrived."
+text_3 = "nin2 hao3，qing3 wen4 nin2 lai2 zi4 na3 zuo4 cheng2 shi4？"
+text_4 = "nin2 hao3，qing4 wen3 nin2 lai2 zi4 na4 zuo3 cheng4 shi3？"
+text_5 = "您好，请问您来自哪 zuo4 cheng2 shi4？"
+text_6 = "/həloʊ, meɪ aɪ æsk wɪt�� sɪti juː ɑːr frʌm?/"
+
+# Use audio from ./assets/audio to avoid downloading from the cloud.
+ref_audio_1 = "https://speech-demo.oss-cn-shanghai.aliyuncs.com/moss_tts_demo/tts_readme_demo/reference_zh.wav"
+ref_audio_2 = "https://speech-demo.oss-cn-shanghai.aliyuncs.com/moss_tts_demo/tts_readme_demo/reference_en.m4a"
+
+conversations = [
+    # Direct TTS (no reference)
+    [processor.build_user_message(text=text_1)],
+    [processor.build_user_message(text=text_2)],
+    # Pinyin or IPA input
+    [processor.build_user_message(text=text_3)],
+    [processor.build_user_message(text=text_4)],
+    [processor.build_user_message(text=text_5)],
+    [processor.build_user_message(text=text_6)],
+    # Voice cloning (with reference)
+    [processor.build_user_message(text=text_1, reference=[ref_audio_1])],
+    [processor.build_user_message(text=text_2, reference=[ref_audio_2])],
+    # Duration control
+    [processor.build_user_message(text=text_2, tokens=325)],
+    [processor.build_user_message(text=text_2, tokens=600)],
+]
+
+model = AutoModel.from_pretrained(
+    pretrained_model_name_or_path,
+    trust_remote_code=True,
+    attn_implementation=attn_implementation,
+    torch_dtype=dtype,
+).to(device)
+model.eval()
+
+batch_size = 1
+
+save_dir = Path("inference_root")
+save_dir.mkdir(exist_ok=True, parents=True)
+sample_idx = 0
+with torch.no_grad():
+    for start in range(0, len(conversations), batch_size):
+        batch_conversations = conversations[start : start + batch_size]
+        batch = processor(batch_conversations, mode="generation")
+        input_ids = batch["input_ids"].to(device)
+        attention_mask = batch["attention_mask"].to(device)
+
+        outputs = model.generate(
+            input_ids=input_ids,
+            attention_mask=attention_mask,
+            max_new_tokens=4096,
+        )
+
+        for message in processor.decode(outputs):
+            audio = message.audio_codes_list[0]
+            out_path = save_dir / f"sample{sample_idx}.wav"
+            sample_idx += 1
+            torchaudio.save(out_path, audio.unsqueeze(0), processor.model_config.sampling_rate)
+
+```
+
+For each model’s full usage, please refer to its corresponding model card.
+
+<a id="fine-tuning"></a>
+## Fine-Tuning
+
+Finetuning tutorials are organized by architecture.
+
+Currently available:
+
+- `MossTTSDelay` / `OpenMOSS-Team/MOSS-TTS`: [moss_tts_delay/finetuning/README.md](moss_tts_delay/finetuning/README.md)
+
+Additional architecture-specific finetuning tutorials will be added under their corresponding directories.
+
+## llama.cpp Backend (Torch-Free Inference)
+
+For lightweight or edge deployment, MOSS-TTS supports a **torch-free** inference path using [llama.cpp](https://github.com/ggerganov/llama.cpp) for the Qwen3 backbone and ONNX Runtime / TensorRT for the audio tokenizer. No PyTorch installation required.
+
+### Quick Start
+
+```bash
+# 1. Install (torch-free)
+pip install -e ".[llama-cpp-onnx]"
+
+# 2. Download pre-quantized backbone + embedding/lm_head weights
+huggingface-cli download OpenMOSS-Team/MOSS-TTS-GGUF --local-dir weights/MOSS-TTS-GGUF
+
+# 3. Download ONNX audio tokenizer
+huggingface-cli download OpenMOSS-Team/MOSS-Audio-Tokenizer-ONNX --local-dir weights/MOSS-Audio-Tokenizer-ONNX
+
+# 4. Build the C bridge (one-time, requires llama.cpp compiled from source)
+cd moss_tts_delay/llama_cpp && bash build_bridge.sh /path/to/llama.cpp && cd ../..
+
+# 5. Run inference
+python -m moss_tts_delay.llama_cpp \
+    --config configs/llama_cpp/default.yaml \
+    --text "Hello, world!" --output output.wav
+
+# 6. (Optional) Low-memory mode for 8 GB GPUs — loads/unloads components per stage
+python -m moss_tts_delay.llama_cpp \
+    --config configs/llama_cpp/trt-8gb.yaml \
+    --text "Hello, world!" --output output.wav
+```
+
+### Installation Profiles
+
+| Profile | Install Command | Dependencies | Use Case |
+|---------|----------------|--------------|----------|
+| **Torch-free (ONNX)** | `pip install -e ".[llama-cpp-onnx]"` | numpy, onnxruntime-gpu, tokenizers | Recommended starting point |
+| **Torch-free (TRT)** | `pip install -e ".[llama-cpp-trt]"` | numpy, tensorrt, cuda-python | Maximum audio tokenizer speed (build engines yourself) |
+| **Torch-accelerated** | `pip install -e ".[llama-cpp-onnx,llama-cpp-torch]"` | + torch | GPU-accelerated LM heads (~30x faster) |
+
+> **Want to convert weights yourself?** See the [conversion guide](moss_tts_delay/llama_cpp/conversion/README.md) for step-by-step instructions on extracting, converting, and quantizing MOSS-TTS weights with llama.cpp.
+
+### Model Weights
+
+| Repository | Contents | Download |
+|-----------|----------|----------|
+| [`OpenMOSS-Team/MOSS-TTS-GGUF`](https://huggingface.co/OpenMOSS-Team/MOSS-TTS-GGUF) | Q4_K_M backbone `.gguf`, `embeddings/` (`.npy`), `lm_heads/` (`.npy`), tokenizer | `huggingface-cli download OpenMOSS-Team/MOSS-TTS-GGUF --local-dir weights/MOSS-TTS-GGUF` |
+| [`OpenMOSS-Team/MOSS-Audio-Tokenizer-ONNX`](https://huggingface.co/OpenMOSS-Team/MOSS-Audio-Tokenizer-ONNX) | Encoder & decoder ONNX models | `huggingface-cli download OpenMOSS-Team/MOSS-Audio-Tokenizer-ONNX --local-dir weights/MOSS-Audio-Tokenizer-ONNX` |
+
+> **Note:** We do **not** provide pre-built TensorRT engines, as they are tied to your specific GPU and TensorRT version. To use TRT, build engines from the ONNX models yourself — see `moss_audio_tokenizer/trt/build_engine.sh`.
+
+### Configuration
+
+Four pre-built configs are provided in `configs/llama_cpp/`:
+
+- `default.yaml` — ONNX audio + GGUF backbone (recommended start)
+- `trt.yaml` — TensorRT audio + GGUF backbone (max throughput, user-built engines)
+- `trt-8gb.yaml` — Low-memory mode for 8 GB GPUs (staged loading, TRT audio)
+- `cpu-only.yaml` — fully CPU-based (no GPU required)
+
+Key config options:
+- `heads_backend: auto | numpy | torch` — LM heads computation backend
+- `audio_backend: onnx | trt | torch` — audio tokenizer backend
+- `low_memory: true | false` — staged loading for limited VRAM (loads/unloads encoder, backbone, decoder per stage)
+- `kv_cache_type_k / kv_cache_type_v` — KV cache quantization (e.g. `q8_0`, `q4_0`) to reduce VRAM
+- `flash_attn: auto | enabled | disabled` — flash attention for lower peak VRAM during prefill
+
+For full documentation, see [moss_tts_delay/llama_cpp/README.md](moss_tts_delay/llama_cpp/README.md).
+
+## Evaluation
+
+This section summarizes the **family‑level evaluation highlights** for MOSS‑TTS and MOSS‑VoiceGenerator. For full details, see each model’s model card.
+
+### MOSS‑TTS
+MOSS‑TTS achieved state‑of‑the‑art results on the open‑source zero‑shot TTS benchmark `Seed‑TTS‑eval`, surpassing all open‑source models and rivaling leading closed‑source systems.
+
+| Model | Params | Open‑source | EN WER (%) ↓ | EN SIM (%) ↑ | ZH CER (%) ↓ | ZH SIM (%) ↑ |
+|---|---:|:---:|---:|---:|---:|---:|
+| DiTAR | 0.6B | ❌ | 1.69 | 73.5 | 1.02 | 75.3 |
+| FishAudio‑S1 | 4B | ❌ | 1.72 | 62.57 | 1.22 | 72.1 |
+| Seed‑TTS |  | ❌ | 2.25 | 76.2 | 1.12 | 79.6 |
+| MiniMax‑Speech |  | ❌ | 1.65 | 69.2 | 0.83 | 78.3 |
+|  |  |  |  |  |  |  |
+| CosyVoice | 0.3B | ✅ | 4.29 | 60.9 | 3.63 | 72.3 |
+| CosyVoice2 | 0.5B | ✅ | 3.09 | 65.9 | 1.38 | 75.7 |
+| CosyVoice3 | 0.5B | ✅ | 2.02 | 71.8 | 1.16 | 78 |
+| CosyVoice3 | 1.5B | ✅ | 2.22 | 72 | 1.12 | 78.1 |
+| F5‑TTS | 0.3B | ✅ | 2 | 67 | 1.53 | 76 |
+| SparkTTS | 0.5B | ✅ | 3.14 | 57.3 | 1.54 | 66 |
+| FireRedTTS | 0.5B | ✅ | 3.82 | 46 | 1.51 | 63.5 |
+| FireRedTTS‑2 | 1.5B | ✅ | 1.95 | 66.5 | 1.14 | 73.6 |
+| Qwen2.5‑Omni | 7B | ✅ | 2.72 | 63.2 | 1.7 | 75.2 |
+| FishAudio‑S1‑mini | 0.5B | ✅ | 1.94 | 55 | 1.18 | 68.5 |
+| IndexTTS2 | 1.5B | ✅ | 2.23 | 70.6 | 1.03 | 76.5 |
+| VibeVoice | 1.5B | ✅ | 3.04 | 68.9 | 1.16 | 74.4 |
+| HiggsAudio‑v2 | 3B | ✅ | 2.44 | 67.7 | 1.5 | 74 |
+| VoxCPM | 0.5B | ✅ | 1.85 | 72.9 | **0.93** | 77.2 |
+| Qwen3‑TTS | 0.6B | ✅ | 1.68 | 70.39 | 1.23 | 76.4 |
+| Qwen3‑TTS | 1.7B | ✅ | **1.5** | 71.45 | 1.33 | 76.72 |
+| GLM-TTS | 1.5B | ✅ | 2.23 | 67.2 | 1.03 | 76.1 |
+| GLM-TTS-RL | 1.5B | ✅ | 1.91 | 68.1 | 0.89 | 76.4 |
+|  |  |  |  |  |  |  |
+| **MossTTSDelay** | **8B** | ✅ | 1.79 | 71.46 | 1.32 | 77.05 |
+| **MossTTSLocal** | **1.7B** | ✅ | 1.85 | **73.42** | 1.2 | **78.82** |
+
+### MOSS‑TTSD
+
+#### Objective Evaluation
+We evaluate MOSS‑TTSD-v1.0 using three objective metrics: Speaker Attribution Accuracy (ACC), Speaker Similarity (SIM), and Word Error Rate (WER). Benchmarked against multiple open-source and closed-source models, the results show that MOSS‑TTSD-v1.0 consistently achieves either the best or second-best performance.
+
+| Model | ZH - SIM | ZH - ACC | ZH - WER | EN - SIM | EN - ACC | EN - WER |
+| :--- | :---: | :---: | :---: | :---: | :---: | :---: |
+| **Comparison with Open-Source Models** | | | | | | |
+| **MOSS-TTSD-v1.0** | **0.7949** | **0.9587** | **0.0485** | **0.7326** | **0.9626** | 0.0988 |
+| MOSS-TTSD-v0.7 | 0.7423 | 0.9391 | 0.0517 | 0.6743 | 0.9266 | 0.1612 |
+| Vibevoice 7B | 0.7590 | 0.9222 | 0.0570 | 0.7140 | 0.9554 | **0.0946** |
+| Vibevoice 1.5 B | 0.7415 | 0.8798 | 0.0818 | 0.6961 | 0.9353 | 0.1133 |
+| FireRedTTS2 | 0.7383 | 0.9022 | 0.0768 | - | - | - |
+| Higgs Audio V2 | - | - | - | 0.6860 | 0.9025 | 0.2131 |
+| **Comparison with Proprietary Models** | | | | | | |
+| **MOSS-TTSD-v1.0 (elevenlabs_voice)** | **0.8165** | **0.9736** | 0.0391 | **0.7304** | **0.9565** | 0.1005 |
+| Eleven V3 | 0.6970 | 0.9653 | **0.0363** | 0.6730 | 0.9498 | **0.0824** |
+| | | | | | | |
+| **MOSS-TTSD-v1.0 (gemini_voice)** | - | - | - | **0.7893** | **0.9655** | 0.0984 |
+| gemini-2.5-pro-preview-tts | - | - | - | 0.6786 | 0.9537 | **0.0859** |
+| gemini-2.5-flash-preview-tts | - | - | - | 0.7194 | 0.9511 | 0.0871 |
+| | | | | | | |
+| **MOSS-TTSD-v1.0 (doubao_voice)** | **0.8226** | **0.9630** | 0.0571 | - | - | - |
+| Doubao_Podcast | 0.8034 | 0.9606 | **0.0472** | - | - | - |
+
+#### Subjective Evaluation
+For open-source models, annotators are asked to score each sample pair in terms of speaker attribution accuracy, voice similarity, prosody, and overall quality. Following the methodology of the LMSYS Chatbot Arena, we compute Elo ratings and confidence intervals for each dimension.
+![alt text](assets/VS_Open-Source_Models.jpg)
+
+For closed-source models, annotators are only asked to choose the overall preferred one in each pair, and we compute the win rate accordingly.
+![alt text](assets/VS_Proprietary_Models.png)
+
+
+### MOSS‑VoiceGenerator
+MOSS‑VoiceGenerator demonstrates strong subjective preference across **overall preference**, **instruction following**, and **naturalness**.
+
+<p align="center">
+  <img src="./assets/moss_voice_generator_winrate.png" width="70%" />
+</p>
+
+## MOSS-Audio-Tokenizer
+
+<a id="mat-intro"></a>
+### Introduction
+**MOSS-Audio-Tokenizer** serves as the unified discrete audio interface for the entire MOSS-TTS Family. It is based on the **Cat** (**C**ausal **A**udio **T**okenizer with **T**ransformer) architecture—a 1.6-billion-parameter, "CNN-free" homogeneous audio tokenizer built entirely from Causal Transformer blocks.
+
+- **Unified Discrete Bridge**: It acts as the shared backbone for MOSS-TTS, MOSS-TTSD, MOSS-VoiceGenerator, MOSS-SoundEffect, and MOSS-TTS-Realtime, providing a consistent audio representation across the family.
+- **Extreme Compression & High Fidelity**: It compresses 24kHz raw audio into a remarkably low frame rate of 12.5Hz. Utilizing a 32-layer Residual Vector Quantizer (RVQ), it supports high-fidelity reconstruction across variable bitrates from 0.125kbps to 4kbps.
+- **Massive-Scale General Audio Training**: Trained from scratch on 3 million hours of diverse data (speech, sound effects, and music), the model achieves state-of-the-art reconstruction among open source audio tokenizers.
+- **Native Streaming Design**: The pure Causal Transformer architecture is specifically designed for scalability and low-latency streaming inference, enabling real-time production workflows.
+
+To learn more about setup, advanced usage, and evaluation metrics, please visit the [MOSS-Audio-Tokenizer Repository](https://github.com/OpenMOSS/MOSS-Audio-Tokenizer)
+
+<p align="center">
+  <img src="./assets/arch_moss_audio_tokenizer.png" alt="MOSS Audio Tokenizer architecture" width="100%" />
+  Architecture of MOSS Audio Tokenizer
+</p>
+
+### Model Weights
+
+| Model | Hugging Face | ModelScope |
+|:-----:|:------------:|:----------:|
+| **MOSS-Audio-Tokenizer** | [![Hugging Face](https://img.shields.io/badge/Huggingface-Model-orange?logo=huggingface)](https://huggingface.co/OpenMOSS-Team/MOSS-Audio-Tokenizer) | [![ModelScope](https://img.shields.io/badge/ModelScope-Model-lightgrey?logo=modelscope)](https://modelscope.cn/models/openmoss/MOSS-Audio-Tokenizer) |
+
+### Objective Reconstruction Evaluation
+
+We compare **MOSS Audio Tokenizer** with open-source audio tokenizers on the LibriSpeech test-clean subset using SIM, STOI, PESQ-NB, and PESQ-WB. Bitrate is controlled by varying the number of RVQ codebooks during decoding, and MOSS Audio Tokenizer leads reconstruction quality among open-source audio tokenizers at comparable 0–4 kbps bitrates.
+
+<p align="center">
+  <img src="./assets/evaluation_moss_audio_tokenizer.png" alt="LibriSpeech objective metrics for audio tokenizers" width="90%" />
+</p>
+
+## LICENSE
+
+Models in MOSS-TTS Family are licensed under the Apache License 2.0.
+
+## Citation
+
+```bibtex
+```
+
+## Star History
+
+[![Star History Chart](https://api.star-history.com/svg?repos=OpenMOSS/MOSS-TTS&type=date&legend=top-left)](https://www.star-history.com/#OpenMOSS/MOSS-TTS&type=date&legend=top-left)

README_zh.md

@@ -0,0 +1,534 @@
+# MOSS-TTS 家族
+
+
+
+<br>
+
+<p align="center" style="display:flex; justify-content:center; align-items:center; gap:24px;">
+  <img src="./assets/OpenMOSS_Logo.png" height="80" style="display:block; transform: translateY(0px);" />
+  <img src="./assets/mosi-logo.png" height="50" style="display:block; transform: translateY(-8px);" />
+</p>
+
+
+
+<div align="center">
+  <a href="https://clawhub.ai/luogao2333/moss-tts-voice"><img src="https://img.shields.io/badge/🦞_OpenClaw-Skills-8A2BE2" alt="OpenClaw"></a>
+  <a href="https://huggingface.co/collections/OpenMOSS-Team/moss-tts"><img src="https://img.shields.io/badge/Huggingface-Models-orange?logo=huggingface&amp"></a>
+  <a href="https://modelscope.cn/collections/OpenMOSS-Team/MOSS-TTS"><img src="https://img.shields.io/badge/ModelScope-Models-lightgrey?logo=modelscope&amp"></a>
+  <a href="https://mosi.cn/#models"><img src="https://img.shields.io/badge/Blog-View-blue?logo=internet-explorer&amp"></a>
+  <a href="https://github.com/OpenMOSS/MOSS-TTS"><img src="https://img.shields.io/badge/Arxiv-Coming%20soon-red?logo=arxiv&amp"></a>
+
+  <a href="https://studio.mosi.cn"><img src="https://img.shields.io/badge/AIStudio-Try-green?logo=internet-explorer&amp"></a>
+  <a href="https://studio.mosi.cn/docs/moss-tts"><img src="https://img.shields.io/badge/API-Docs-00A3FF?logo=fastapi&amp"></a>
+  <a href="https://x.com/Open_MOSS"><img src="https://img.shields.io/badge/Twitter-Follow-black?logo=x&amp"></a>
+  <a href="https://discord.gg/fvm5TaWjU3"><img src="https://img.shields.io/badge/Discord-Join-5865F2?logo=discord&amp"></a>
+  <a href="./assets/wechat.jpg"><img src="https://img.shields.io/badge/WeChat-Join-07C160?logo=wechat&amp;logoColor=white" alt="WeChat"></a>
+  
+</div>
+
+
+[English](README.md) | [简体中文](README_zh.md)
+
+
+MOSS‑TTS 家族是由 [MOSI.AI](https://mosi.cn/#hero) 与 [OpenMOSS 团队](https://www.open-moss.com/) 推出的开源 **语音与声音生成模型家族**。该系列面向 **高保真**、**高表现力** 与 **复杂真实场景** 设计，覆盖稳定长文本语音、多说话人对话、音色/角色设计、环境音效以及实时流式 TTS 等能力。
+
+<a id="news"></a>
+## 新闻
+* 2026.3.10：⚡️ 大幅优化了 llama.cpp 推理管线的显存占用。现在 8B 模型可以运行在 8GB 显存的 GPU 上！
+* 2026.3.4：新增 **无 PyTorch 推理** 支持 — 通过 [llama.cpp](https://github.com/ggerganov/llama.cpp) + ONNX Runtime 实现端侧轻量部署。量化 GGUF 权重发布于 [`OpenMOSS-Team/MOSS-TTS-GGUF`](https://huggingface.co/OpenMOSS-Team/MOSS-TTS-GGUF)，ONNX 音频编解码器发布于 [`OpenMOSS-Team/MOSS-Audio-Tokenizer-ONNX`](https://huggingface.co/OpenMOSS-Team/MOSS-Audio-Tokenizer-ONNX)。详见 [llama.cpp 后端](#llamacpp-后端无-pytorch-推理)。
+* 2026.3.4：🎉 我们在 🦞 龙虾 的 [ClawHub](https://clawhub.ai) 平台上架了 MOSS-TTS skills：[feishu-voice-tts](https://clawhub.ai/helloeveryworlds/feishu-voice-tts) 与 [moss-tts-voice](https://clawhub.ai/luogao2333/moss-tts-voice)。
+* 2026.2.10：🎉🎉🎉 我们已发布 [MOSS-TTS Family](https://huggingface.co/collections/OpenMOSS-Team/moss-tts)。更多详情请查看我们的 [Blog](https://mosi.cn/#models)！我们的 Huggingface Space 在这里：[MOSS-TTS](https://huggingface.co/spaces/OpenMOSS-Team/MOSS-TTS), [MOSS-TTSD-v1.0](https://huggingface.co/spaces/OpenMOSS-Team/MOSS-TTSD-v1.0), [MOSS-VoiceGenerator](https://huggingface.co/spaces/OpenMOSS-Team/MOSS-VoiceGenerator).
+
+## 演示
+
+<div align="center">
+  <video src="https://gist.github.com/user-attachments/assets/fdce9f66-20ec-45e8-9615-89606ae2fbe8" width="70%" poster=""> </video>
+</div>
+
+
+## 目录
+
+- [介绍](#introduction)
+- [模型架构](#architecture)
+- [已发布模型](#released-models)
+- [支持的语言](#supported-languages)
+- [快速开始](#quickstart)
+  - [OpenClaw API Skills](#openclaw-api-skills)
+  - [环境准备](#environment-setup)
+  - [（可选）安装 FlashAttention 2](#optional-install-flashattention-2)
+  - [基础用法](#moss-tts-basic-usage)
+  - [微调](#fine-tuning)
+- [llama.cpp 后端（无 PyTorch 推理）](#llamacpp-后端无-pytorch-推理)
+- [评测](#evaluation)
+  - [MOSS-TTS 评测](#eval-moss-tts)
+  - [MOSS-TTSD 评测](#eval-moss-ttsd)
+  - [MOSS-VoiceGenerator 评测](#eval-moss-voicegenerator)
+- [语音编解码器](#audio-tokenizer)
+  - [介绍](#audio-tokenizer-intro)
+  - [模型权重](#model-weights)
+  - [重建质量客观评测](#重建质量客观评测)
+
+
+<a id="introduction"></a>
+## 介绍
+
+<p align="center">
+  <img src="./assets/moss_tts_family.jpeg" width="85%" />
+</p>
+
+当一段音频需要 **听起来像真实的人类**、**准确发音**、**在不同内容间切换说话风格**、**稳定持续数十分钟**，并且 **支持对话、角色扮演与实时交互** 时，单一 TTS 模型往往不足以胜任。**MOSS‑TTS 家族**将工作流拆分为 5 个可独立使用、亦可组合成完整管线的量产级模型。
+
+- **MOSS‑TTS**：MOSS‑TTS 是家族中的旗舰量产级 TTS 基础模型，**核心能力是高保真以及最优性能的零样本语音克隆**，支持**长文本长语音生成**、**拼音、音标与时长精细控制**，以及**多语种/中英混合合成**。它可作为大规模旁白、配音和语音产品的核心底座。
+- **MOSS‑TTSD**：MOSS‑TTSD 是对话语音生成模型，用于生成高表现力、多说话人、超长连续对话的音频。本次我们更新了全新的**v1.0版本**，相比于0.7版本，它在音色相似度，说话人切换准确率，词错误率等**客观指标上取得了业界最优的性能**，在竞技场主观评测中，也**战胜了豆包、Gemini2.5-pro**等顶尖闭源模型。详情请访问 [MOSS-TTSD 仓库](https://github.com/OpenMOSS/MOSS-TTSD)。
+- **MOSS‑VoiceGenerator**：MOSS‑VoiceGenerator 是开源音色设计模型，可从文本风格指令直接生成多样的说话人音色或风格，**无需参考音频**。它统一音色设计、风格控制与内容合成，可独立创作，也可作为下游 TTS 的音色设计层。模型性能在**竞技场评分上超过了其余等顶尖音色设计模型**。
+- **MOSS‑TTS‑Realtime**：MOSS‑TTS‑Realtime 是面向实时语音智能体的多轮上下文感知实时 TTS 模型。它结合多轮对话中的文本与历史语音信号进行低时延增量合成，使多轮回复保持连贯、自然且音色一致。**非常适合搭配文本模型构建低时延语音智能体**。
+- **MOSS‑SoundEffect**：MOSS‑SoundEffect 是面向内容制作的**音效生成**模型，具备广泛类别覆盖与可控时长能力。它能根据文本指令生成自然环境、城市场景、生物、人类动作与类音乐片段等音频，适用于影视、游戏、交互体验和数据合成。
+
+<a id="architecture"></a>
+## 模型架构
+
+我们在统一训练/评测框架下将 **MossTTSDelay** 与 **MossTTSLocal** 作为互补基线：**Delay** 更强调长上下文稳定性、推理速度与工程可用性，**Local** 更强调轻量灵活和面向流式场景的客观指标表现。二者共同提供可复现、可对比的落地与研究参考。
+
+**MossTTSRealtime** 不是第三个对比基线，而是面向语音智能体的能力型设计。它同时利用历史文本与用户语音声学信息建模多轮上下文，以低时延流式合成保持回复连贯和音色一致。
+
+
+| 架构  | 核心机制 | 架构细节 |
+|---|---|---|
+| `MossTTSDelay` |  多头并行 RVQ 预测，结合延迟模式调度 | [![Arch Details](https://img.shields.io/badge/Model%20Card-View-blue?logo=markdown)](moss_tts_delay/README.md) |
+| `MossTTSLocal` | 基于深度 Transformer 的时间同步 RVQ 模块 | [![Arch Details](https://img.shields.io/badge/Model%20Card-View-blue?logo=markdown)](moss_tts_local/README.md) |
+| `MossTTSRealtime` | 用于实时合成的分层文本-音频输入 | [![Arch Details](https://img.shields.io/badge/Model%20Card-View-blue?logo=markdown)](moss_tts_realtime/README.md) |
+
+<a id="released-models"></a>
+## 模型概览
+
+| Model | Architecture | Size | Model Card | Hugging Face | ModelScope |
+|---|---|---:|---|---|---|
+| **MOSS-TTS** | `MossTTSDelay` | 8B | [![Model Card](https://img.shields.io/badge/Model%20Card-View-blue?logo=markdown)](docs/moss_tts_model_card.md) | [![Hugging Face](https://img.shields.io/badge/Huggingface-Model-orange?logo=huggingface)](https://huggingface.co/OpenMOSS-Team/MOSS-TTS) | [![ModelScope](https://img.shields.io/badge/ModelScope-Model-lightgrey?logo=modelscope)](https://modelscope.cn/models/openmoss/MOSS-TTS) |
+|  | `MossTTSLocal` | 1.7B | [![Model Card](https://img.shields.io/badge/Model%20Card-View-blue?logo=markdown)](docs/moss_tts_model_card.md) | [![Hugging Face](https://img.shields.io/badge/Huggingface-Model-orange?logo=huggingface)](https://huggingface.co/OpenMOSS-Team/MOSS-TTS-Local-Transformer) | [![ModelScope](https://img.shields.io/badge/ModelScope-Model-lightgrey?logo=modelscope)](https://modelscope.cn/models/openmoss/MOSS-TTS-Local-Transformer) |
+| **MOSS‑TTSD‑V1.0** | `MossTTSDelay` | 8B | [![Model Card](https://img.shields.io/badge/Model%20Card-View-blue?logo=markdown)](docs/moss_ttsd_model_card.md) | [![Hugging Face](https://img.shields.io/badge/Huggingface-Model-orange?logo=huggingface)](https://huggingface.co/OpenMOSS-Team/MOSS-TTSD-v1.0) | [![ModelScope](https://img.shields.io/badge/ModelScope-Model-lightgrey?logo=modelscope)](https://modelscope.cn/models/openmoss/MOSS-TTSD-v1.0) |
+| **MOSS‑VoiceGenerator** | `MossTTSDelay` | 1.7B | [![Model Card](https://img.shields.io/badge/Model%20Card-View-blue?logo=markdown)](docs/moss_voice_generator_model_card.md) | [![Hugging Face](https://img.shields.io/badge/Huggingface-Model-orange?logo=huggingface)](https://huggingface.co/OpenMOSS-Team/MOSS-VoiceGenerator) | [![ModelScope](https://img.shields.io/badge/ModelScope-Model-lightgrey?logo=modelscope)](https://modelscope.cn/models/openmoss/MOSS-VoiceGenerator) |
+| **MOSS‑SoundEffect** | `MossTTSDelay` | 8B | [![Model Card](https://img.shields.io/badge/Model%20Card-View-blue?logo=markdown)](docs/moss_sound_effect_model_card.md) | [![Hugging Face](https://img.shields.io/badge/Huggingface-Model-orange?logo=huggingface)](https://huggingface.co/OpenMOSS-Team/MOSS-SoundEffect) | [![ModelScope](https://img.shields.io/badge/ModelScope-Model-lightgrey?logo=modelscope)](https://modelscope.cn/models/openmoss/MOSS-SoundEffect) |
+| **MOSS‑TTS‑Realtime** | `MossTTSRealtime` | 1.7B | [![Model Card](https://img.shields.io/badge/Model%20Card-View-blue?logo=markdown)](docs/moss_tts_realtime_model_card.md) | [![Hugging Face](https://img.shields.io/badge/Huggingface-Model-orange?logo=huggingface)](https://huggingface.co/OpenMOSS-Team/MOSS-TTS-Realtime) | [![ModelScope](https://img.shields.io/badge/ModelScope-Model-lightgrey?logo=modelscope)](https://modelscope.cn/models/openmoss/MOSS-TTS-Realtime) |
+
+<a id="supported-languages"></a>
+
+## 支持的语言
+
+MOSS-TTS、MOSS-TTSD 和 MOSS-TTS-Realtime 目前支持 **20 种语言**：
+
+| Language | Code | Flag | Language | Code | Flag | Language | Code | Flag |
+|---|---|---|---|---|---|---|---|---|
+| 中文 | zh | 🇨🇳 | 英语 | en | 🇺🇸 | 德语 | de | 🇩🇪 |
+| 西班牙语 | es | 🇪🇸 | 法语 | fr | 🇫🇷 | 日语 | ja | 🇯🇵 |
+| 意大利语 | it | 🇮🇹 | 匈牙利语 | hu | 🇭🇺 | 韩语 | ko | 🇰🇷 |
+| 俄语 | ru | 🇷🇺 | 波斯语（法尔西语） | fa | 🇮🇷 | 阿拉伯语 | ar | 🇸🇦 |
+| 波兰语 | pl | 🇵🇱 | 葡萄牙语 | pt | 🇵🇹 | 捷克语 | cs | 🇨🇿 |
+| 丹麦语 | da | 🇩🇰 | 瑞典语 | sv | 🇸🇪 | | | |
+| 希腊语 | el | 🇬🇷 | 土耳其语 | tr | 🇹🇷 |  |  |  |
+
+
+
+<a id="quickstart"></a>
+## 快速开始
+
+### OpenClaw API Skills
+
+我们在🦞 龙虾 的 [ClawHub](https://clawhub.ai) 平台上架了 MOSS-TTS skills。API Key 可在 [MOSI AI Studio](https://studio.mosi.cn) 获取。
+
+| Skill | 说明 | 安装命令 |
+|---|---|---|
+| [`feishu-voice-tts`](https://clawhub.ai/helloeveryworlds/feishu-voice-tts) | 在飞书发送语音消息 | `clawhub install feishu-voice-tts` |
+| [`moss-tts-voice`](https://clawhub.ai/luogao2333/moss-tts-voice) | 调用 MOSS-TTS API 生成语音 | `clawhub install moss-tts-voice` |
+
+<a id="environment-setup"></a>
+### 环境准备
+
+建议使用干净的 Python 环境。
+
+#### 使用 Conda
+
+```bash
+conda create -n moss-tts python=3.12 -y
+conda activate moss-tts
+```
+
+安装全部依赖：
+
+```bash
+git clone https://github.com/OpenMOSS/MOSS-TTS.git
+cd MOSS-TTS
+pip install --extra-index-url https://download.pytorch.org/whl/cu128 -e ".[torch-runtime]"
+```
+
+#### 使用 `uv`
+
+```bash
+# 请先安装 uv：https://docs.astral.sh/uv/getting-started/installation/
+git clone https://github.com/OpenMOSS/MOSS-TTS.git
+cd MOSS-TTS
+uv venv --python 3.12 .venv
+source .venv/bin/activate
+uv pip install --torch-backend cu128 -e ".[torch-runtime]"
+```
+<a id="optional-install-flashattention-2"></a>
+#### （可选）安装 FlashAttention 2
+
+如果你的硬件支持，可以安装 FlashAttention 2 以提升速度并降低显存占用。
+
+如果你使用 Conda/pip：
+
+```bash
+pip install --extra-index-url https://download.pytorch.org/whl/cu128 -e ".[torch-runtime,flash-attn]"
+```
+
+如果机器内存较小、CPU 核数较多，可以限制并行编译数：
+
+```bash
+MAX_JOBS=4 pip install --extra-index-url https://download.pytorch.org/whl/cu128 -e ".[torch-runtime,flash-attn]"
+```
+
+如果你使用 `uv`：
+
+```bash
+uv pip install --torch-backend cu128 -e ".[torch-runtime,flash-attn]"
+```
+
+如果机器内存较小、CPU 核心较多，可以限制并行编译数：
+
+```bash
+MAX_JOBS=4 uv pip install --torch-backend cu128 -e ".[torch-runtime,flash-attn]"
+```
+
+说明：
+- 依赖统一在 `pyproject.toml` 中管理，当前固定了 `torch==2.9.1+cu128` 和 `torchaudio==2.9.1+cu128`。
+- `uv` 方案中使用 `--torch-backend cu128`，由 uv 处理 PyTorch CUDA 轮子来源，同时其余依赖仍使用默认安全索引策略解析。
+- 如果需要其他后端，可将 `cu128` 替换为目标后端（例如 `cpu`、`cu126`）。
+- 如果 FlashAttention 2 编译失败，可以跳过，直接使用默认 attention 后端。
+- FlashAttention 2 仅支持部分 GPU，通常搭配 `torch.float16` 或 `torch.bfloat16` 使用。
+
+
+<a id="moss-tts-basic-usage"></a>
+### MOSS‑TTS 基础用法
+
+如果你更希望使用 Gradio 界面，我们为 4 个主模型提供了对应脚本：
+
+| Model | Script | 
+|---|---|
+| MOSS-TTS | [clis/moss_tts_app.py](clis/moss_tts_app.py) |
+| MOSS-TTSD | [clis/moss_ttsd_app.py](clis/moss_ttsd_app.py) | 
+| MOSS-VoiceGenerator | [clis/moss_voice_generator_app.py](clis/moss_voice_generator_app.py) | 
+| MOSS-SoundEffect | [clis/moss_sound_effect_app.py](clis/moss_sound_effect_app.py) |
+
+MOSS-TTS-Realtime 的 Gradio demo 请直接参考 [MOSS-TTS-Realtime Model Card](docs/moss_tts_realtime_model_card.md)
+
+```python
+from pathlib import Path
+import importlib.util
+import torch
+import torchaudio
+from transformers import AutoModel, AutoProcessor
+# Disable the broken cuDNN SDPA backend
+torch.backends.cuda.enable_cudnn_sdp(False)
+# Keep these enabled as fallbacks
+torch.backends.cuda.enable_flash_sdp(True)
+torch.backends.cuda.enable_mem_efficient_sdp(True)
+torch.backends.cuda.enable_math_sdp(True)
+
+
+pretrained_model_name_or_path = "OpenMOSS-Team/MOSS-TTS"
+device = "cuda" if torch.cuda.is_available() else "cpu"
+dtype = torch.bfloat16 if device == "cuda" else torch.float32
+
+def resolve_attn_implementation() -> str:
+    # Prefer FlashAttention 2 when package + device conditions are met.
+    if (
+        device == "cuda"
+        and importlib.util.find_spec("flash_attn") is not None
+        and dtype in {torch.float16, torch.bfloat16}
+    ):
+        major, _ = torch.cuda.get_device_capability()
+        if major >= 8:
+            return "flash_attention_2"
+
+    # CUDA fallback: use PyTorch SDPA kernels.
+    if device == "cuda":
+        return "sdpa"
+
+    # CPU fallback.
+    return "eager"
+
+
+attn_implementation = resolve_attn_implementation()
+print(f"[INFO] Using attn_implementation={attn_implementation}")
+
+processor = AutoProcessor.from_pretrained(
+    pretrained_model_name_or_path,
+    trust_remote_code=True,
+)
+processor.audio_tokenizer = processor.audio_tokenizer.to(device)
+
+text_1 = "亲爱的你，\n你好呀。\n\n今天，我想用最认真、最温柔的声音，对你说一些重要的话。\n这些话，像一颗小小的星星，希望能在你的心里慢慢发光。\n\n首先，我想祝你——\n每天都能平平安安、快快乐乐。\n\n希望你早上醒来的时候，\n窗外有光，屋子里很安静，\n你的心是轻轻的，没有着急，也没有害怕。\n\n希望你吃饭的时候胃口很好，\n走路的时候脚步稳稳，\n晚上睡觉的时候，能做一个又一个甜甜的梦。\n\n我希望你能一直保持好奇心。\n对世界充满问题，\n对天空、星星、花草、书本和故事感兴趣。\n当你问“为什么”的时候，\n希望总有人愿意认真地听你说话。\n\n我也希望你学会温柔。\n温柔地对待朋友，\n温柔地对待小动物，\n也温柔地对待自己。\n\n如果有一天你犯了错，\n请不要太快责怪自己，\n因为每一个认真成长的人，\n都会在路上慢慢学会更好的方法。\n\n愿你拥有勇气。\n当你站在陌生的地方时，\n当你第一次举手发言时，\n当你遇到困难、感到害怕的时候，\n希望你能轻轻地告诉自己：\n“我可以试一试。”\n\n就算没有一次成功，也没有关系。\n失败不是坏事，\n它只是告诉你，你正在努力。\n\n我希望你学会分享快乐。\n把开心的事情告诉别人，\n把笑声送给身边的人，\n因为快乐被分享的时候，\n会变得更大、更亮。\n\n如果有一天你感到难过，\n我希望你知道——\n难过并不丢脸，\n哭泣也不是软弱。\n\n愿你能找到一个安全的地方，\n慢慢把心里的话说出来，\n然后再一次抬起头，看见希望。\n\n我还希望你能拥有梦想。\n这个梦想也许很大，\n也许很小，\n也许现在还说不清楚。\n\n没关系。\n梦想会和你一起长大，\n在时间里慢慢变得清楚。\n\n最后，我想送你一个最最重要的祝福：\n\n愿你被世界温柔对待，\n也愿你成为一个温柔的人。\n\n愿你的每一天，\n都值得被记住，\n都值得被珍惜。\n\n亲爱的你，\n请记住，\n你是独一无二的，\n你已经很棒了，\n而你的未来，\n一定会慢慢变得闪闪发光。\n\n祝你健康、勇敢、幸福，\n祝你永远带着笑容向前走。"
+text_2 = "We stand on the threshold of the AI era.\nArtificial intelligence is no longer just a concept in laboratories, but is entering every industry, every creative endeavor, and every decision. It has learned to see, hear, speak, and think, and is beginning to become an extension of human capabilities. AI is not about replacing humans, but about amplifying human creativity, making knowledge more equitable, more efficient, and allowing imagination to reach further. A new era, jointly shaped by humans and intelligent systems, has arrived."
+text_3 = "nin2 hao3，qing3 wen4 nin2 lai2 zi4 na3 zuo4 cheng2 shi4？"
+text_4 = "nin2 hao3，qing4 wen3 nin2 lai2 zi4 na4 zuo3 cheng4 shi3？"
+text_5 = "您好，请问您来自哪 zuo4 cheng2 shi4？"
+text_6 = "/həloʊ, meɪ aɪ æsk wɪtʃ sɪti juː ɑːr frʌm?/"
+
+# Use audio from ./assets/audio to avoid downloading from the cloud.
+ref_audio_1 = "https://speech-demo.oss-cn-shanghai.aliyuncs.com/moss_tts_demo/tts_readme_demo/reference_zh.wav"
+ref_audio_2 = "https://speech-demo.oss-cn-shanghai.aliyuncs.com/moss_tts_demo/tts_readme_demo/reference_en.m4a"
+
+conversations = [
+    # Direct TTS (no reference)
+    [processor.build_user_message(text=text_1)],
+    [processor.build_user_message(text=text_2)],
+    # Pinyin or IPA input
+    [processor.build_user_message(text=text_3)],
+    [processor.build_user_message(text=text_4)],
+    [processor.build_user_message(text=text_5)],
+    [processor.build_user_message(text=text_6)],
+    # Voice cloning (with reference)
+    [processor.build_user_message(text=text_1, reference=[ref_audio_1])],
+    [processor.build_user_message(text=text_2, reference=[ref_audio_2])],
+    # Duration control
+    [processor.build_user_message(text=text_2, tokens=325)],
+    [processor.build_user_message(text=text_2, tokens=600)],
+]
+
+model = AutoModel.from_pretrained(
+    pretrained_model_name_or_path,
+    trust_remote_code=True,
+    attn_implementation=attn_implementation,
+    torch_dtype=dtype,
+).to(device)
+model.eval()
+
+batch_size = 1
+
+save_dir = Path("inference_root")
+save_dir.mkdir(exist_ok=True, parents=True)
+sample_idx = 0
+with torch.no_grad():
+    for start in range(0, len(conversations), batch_size):
+        batch_conversations = conversations[start : start + batch_size]
+        batch = processor(batch_conversations, mode="generation")
+        input_ids = batch["input_ids"].to(device)
+        attention_mask = batch["attention_mask"].to(device)
+
+        outputs = model.generate(
+            input_ids=input_ids,
+            attention_mask=attention_mask,
+            max_new_tokens=4096,
+        )
+
+        for message in processor.decode(outputs):
+            audio = message.audio_codes_list[0]
+            out_path = save_dir / f"sample{sample_idx}.wav"
+            sample_idx += 1
+            torchaudio.save(out_path, audio.unsqueeze(0), processor.model_config.sampling_rate)
+
+```
+
+各模型的完整使用方式请参考对应的 model card。
+
+<a id="fine-tuning"></a>
+### 微调
+
+微调教程按架构分别组织。
+
+当前已提供：
+
+- `MossTTSDelay` / `OpenMOSS-Team/MOSS-TTS`：见 [moss_tts_delay/finetuning/README_zh.md](moss_tts_delay/finetuning/README_zh.md)
+
+后续其余架构的微调教程也会分别补充到对应目录下。
+
+## llama.cpp 后端（无 PyTorch 推理）
+
+MOSS-TTS 支持使用 [llama.cpp](https://github.com/ggerganov/llama.cpp) 运行 Qwen3 backbone，配合 ONNX Runtime / TensorRT 运行音频编解码器，实现 **完全无 PyTorch 依赖** 的轻量端侧推理。
+
+### 快速开始
+
+```bash
+# 1. 安装（无 PyTorch）
+pip install -e ".[llama-cpp-onnx]"
+
+# 2. 下载预量化 backbone + embedding/lm_head 权重
+huggingface-cli download OpenMOSS-Team/MOSS-TTS-GGUF --local-dir weights/MOSS-TTS-GGUF
+
+# 3. 下载 ONNX 音频编解码器
+huggingface-cli download OpenMOSS-Team/MOSS-Audio-Tokenizer-ONNX --local-dir weights/MOSS-Audio-Tokenizer-ONNX
+
+# 4. 编译 C bridge（一次性，需要 llama.cpp 源码编译）
+cd moss_tts_delay/llama_cpp && bash build_bridge.sh /path/to/llama.cpp && cd ../..
+
+# 5. 推理
+python -m moss_tts_delay.llama_cpp \
+    --config configs/llama_cpp/default.yaml \
+    --text "你好世界！" --output output.wav
+
+# 6. (可选) 针对 8 GB 显存 GPU 的低显存模式 — 按阶段加载/卸载组件
+python -m moss_tts_delay.llama_cpp \
+    --config configs/llama_cpp/trt-8gb.yaml \
+    --text "你好世界！" --output output.wav
+```
+
+### 安装方案
+
+| 方案 | 安装命令 | 依赖 | 适用场景 |
+|------|---------|------|---------|
+| **无 Torch (ONNX)** | `pip install -e ".[llama-cpp-onnx]"` | numpy, onnxruntime-gpu, tokenizers | 推荐入门方案 |
+| **无 Torch (TRT)** | `pip install -e ".[llama-cpp-trt]"` | numpy, tensorrt, cuda-python | 最高音频编解码器性能（需自行编译 engine） |
+| **Torch 加速** | `pip install -e ".[llama-cpp-onnx,llama-cpp-torch]"` | + torch | GPU 加速 LM heads（约 30 倍提速） |
+
+> **想要自行转换权重？** 请参阅 [转换指南](moss_tts_delay/llama_cpp/conversion/README_zh.md)，了解如何使用 llama.cpp 提取、转换和量化 MOSS-TTS 权重。
+
+### 模型权重
+
+| 仓库 | 内容 | 下载命令 |
+|------|------|---------|
+| [`OpenMOSS-Team/MOSS-TTS-GGUF`](https://huggingface.co/OpenMOSS-Team/MOSS-TTS-GGUF) | Q4_K_M backbone `.gguf`、`embeddings/`（`.npy`）、`lm_heads/`（`.npy`）、tokenizer | `huggingface-cli download OpenMOSS-Team/MOSS-TTS-GGUF --local-dir weights/MOSS-TTS-GGUF` |
+| [`OpenMOSS-Team/MOSS-Audio-Tokenizer-ONNX`](https://huggingface.co/OpenMOSS-Team/MOSS-Audio-Tokenizer-ONNX) | Encoder & decoder ONNX 模型 | `huggingface-cli download OpenMOSS-Team/MOSS-Audio-Tokenizer-ONNX --local-dir weights/MOSS-Audio-Tokenizer-ONNX` |
+
+> **注意：** 我们 **不提供** 预编译的 TensorRT engine，因为 TRT engine 与 GPU 架构和 TensorRT 版本强绑定。如需使用 TRT，请从 ONNX 模型自行编译 — 参考 `moss_audio_tokenizer/trt/build_engine.sh`。
+
+### 配置
+
+`configs/llama_cpp/` 中提供了四个预设配置：
+
+- `default.yaml` — ONNX 音频 Tokenizer + GGUF backbone（推荐入门）
+- `trt.yaml` — TensorRT 音频 Tokenizer + GGUF backbone（最大吞吐，需自行编译 engine）
+- `trt-8gb.yaml` — 针对 8 GB 显存 GPU 的低显存模式（分阶段加载，TRT 音频）
+- `cpu-only.yaml` — 纯 CPU 运行（无需 GPU）
+
+关键配置项：
+- `heads_backend: auto | numpy | torch` — LM heads 计算后端
+- `audio_backend: onnx | trt | torch` — 音频编解码器后端
+- `low_memory: true | false` — 针对有限显存的分阶段加载（按阶段加载/卸载 encoder, backbone, decoder）
+- `kv_cache_type_k / kv_cache_type_v` — KV cache 量化（例如 `q8_0`, `q4_0`）以减少显存占用
+- `flash_attn: auto | enabled | disabled` — flash attention 用于降低 prefill 阶段的峰值显存
+
+完整文档请查看 [moss_tts_delay/llama_cpp/README.md](moss_tts_delay/llama_cpp/README.md)。
+
+<a id="evaluation"></a>
+## 评测
+
+本节总结 MOSS‑TTS 与 MOSS‑VoiceGenerator 的 **家族级评测亮点**。完整细节请参见各模型的 model card。
+
+<a id="eval-moss-tts"></a>
+### MOSS‑TTS 评测
+MOSS‑TTS 在开源零样本 TTS 基准 `Seed‑TTS‑eval` 上取得当前最佳结果，超越所有开源模型，并与主流闭源系统相当。
+
+| Model | Params | Open‑source | EN WER (%) ↓ | EN SIM (%) ↑ | ZH CER (%) ↓ | ZH SIM (%) ↑ |
+|---|---:|:---:|---:|---:|---:|---:|
+| DiTAR | 0.6B | ❌ | 1.69 | 73.5 | 1.02 | 75.3 |
+| FishAudio‑S1 | 4B | ❌ | 1.72 | 62.57 | 1.22 | 72.1 |
+| Seed‑TTS |  | ❌ | 2.25 | 76.2 | 1.12 | 79.6 |
+| MiniMax‑Speech |  | ❌ | 1.65 | 69.2 | 0.83 | 78.3 |
+|  |  |  |  |  |  |  |
+| CosyVoice | 0.3B | ✅ | 4.29 | 60.9 | 3.63 | 72.3 |
+| CosyVoice2 | 0.5B | ✅ | 3.09 | 65.9 | 1.38 | 75.7 |
+| CosyVoice3 | 0.5B | ✅ | 2.02 | 71.8 | 1.16 | 78 |
+| CosyVoice3 | 1.5B | ✅ | 2.22 | 72 | 1.12 | 78.1 |
+| F5‑TTS | 0.3B | ✅ | 2 | 67 | 1.53 | 76 |
+| SparkTTS | 0.5B | ✅ | 3.14 | 57.3 | 1.54 | 66 |
+| FireRedTTS | 0.5B | ✅ | 3.82 | 46 | 1.51 | 63.5 |
+| FireRedTTS‑2 | 1.5B | ✅ | 1.95 | 66.5 | 1.14 | 73.6 |
+| Qwen2.5‑Omni | 7B | ✅ | 2.72 | 63.2 | 1.7 | 75.2 |
+| FishAudio‑S1‑mini | 0.5B | ✅ | 1.94 | 55 | 1.18 | 68.5 |
+| IndexTTS2 | 1.5B | ✅ | 2.23 | 70.6 | 1.03 | 76.5 |
+| VibeVoice | 1.5B | ✅ | 3.04 | 68.9 | 1.16 | 74.4 |
+| HiggsAudio‑v2 | 3B | ✅ | 2.44 | 67.7 | 1.5 | 74 |
+| VoxCPM | 0.5B | ✅ | 1.85 | 72.9 | **0.93** | 77.2 |
+| Qwen3‑TTS | 0.6B | ✅ | 1.68 | 70.39 | 1.23 | 76.4 |
+| Qwen3‑TTS | 1.7B | ✅ | **1.5** | 71.45 | 1.33 | 76.72 |
+|  |  |  |  |  |  |  |
+| **MossTTSDelay** | **8B** | ✅ | 1.79 | 71.46 | 1.32 | 77.05 |
+| **MossTTSLocal** | **1.7B** | ✅ | 1.85 | **73.42** | 1.2 | **78.82** |
+
+<a id="eval-moss-ttsd"></a>
+### MOSS‑TTSD 评测
+#### 客观评测
+我们使用三个客观指标来评估 MOSS‑TTSD-v1.0 的性能：说话人归属准确性（ACC）、说话人相似度（SIM）和词错误率（WER）。我们对比了 MOSS‑TTSD-v1.0 与多个开源模型和闭源模型的性能，结果如下，MOSS-TTSD-v1.0 均取得了最优或次优性能。
+
+| Model | ZH - SIM | ZH - ACC | ZH - WER | EN - SIM | EN - ACC | EN - WER |
+| :--- | :---: | :---: | :---: | :---: | :---: | :---: |
+| **Comparison with Open-Source Models** | | | | | | |
+| **MOSS-TTSD-v1.0** | **0.7949** | **0.9587** | **0.0485** | **0.7326** | **0.9626** | 0.0988 |
+| MOSS-TTSD-v0.7 | 0.7423 | 0.9391 | 0.0517 | 0.6743 | 0.9266 | 0.1612 |
+| Vibevoice 7B | 0.7590 | 0.9222 | 0.0570 | 0.7140 | 0.9554 | **0.0946** |
+| Vibevoice 1.5 B | 0.7415 | 0.8798 | 0.0818 | 0.6961 | 0.9353 | 0.1133 |
+| FireRedTTS2 | 0.7383 | 0.9022 | 0.0768 | - | - | - |
+| Higgs Audio V2 | - | - | - | 0.6860 | 0.9025 | 0.2131 |
+| **Comparison with Proprietary Models** | | | | | | |
+| **MOSS-TTSD-v1.0 (elevenlabs_voice)** | **0.8165** | **0.9736** | 0.0391 | **0.7304** | **0.9565** | 0.1005 |
+| Eleven V3 | 0.6970 | 0.9653 | **0.0363** | 0.6730 | 0.9498 | **0.0824** |
+| | | | | | | |
+| **MOSS-TTSD-v1.0 (gemini_voice)** | - | - | - | **0.7893** | **0.9655** | 0.0984 |
+| gemini-2.5-pro-preview-tts | - | - | - | 0.6786 | 0.9537 | **0.0859** |
+| gemini-2.5-flash-preview-tts | - | - | - | 0.7194 | 0.9511 | 0.0871 |
+| | | | | | | |
+| **MOSS-TTSD-v1.0 (doubao_voice)** | **0.8226** | **0.9630** | 0.0571 | - | - | - |
+| Doubao_Podcast | 0.8034 | 0.9606 | **0.0472** | - | - | - |
+
+#### 主观评测
+对于开源模型，标注者会从说话人归属准确性、音色相似度、韵律与整体质量等维度对每个样本对进行评分。遵循 LMSYS Chatbot Arena 的方法，我们计算各维度的 Elo 评分与置信区间。
+![alt text](assets/VS_Open-Source_Models.jpg)
+
+对于闭源模型，标注者只需在每个样本对中选择整体更偏好的一项，并据此计算胜率。
+![alt text](assets/VS_Proprietary_Models.png)
+
+
+<a id="eval-moss-voicegenerator"></a>
+### MOSS‑VoiceGenerator 主观评测
+MOSS‑VoiceGenerator 在 **整体偏好**、**指令遵循** 与 **自然度** 上表现出强主观偏好。
+
+<p align="center">
+  <img src="./assets/moss_voice_generator_winrate.png" width="70%" />
+</p>
+
+<a id="audio-tokenizer"></a>
+## 语音编解码器
+
+<a id="audio-tokenizer-intro"></a>
+### 介绍
+**MOSS-Audio-Tokenizer** 是 MOSS‑TTS 家族的统一离散音频接口，基于 **Cat**（**C**ausal **A**udio **T**okenizer with **T**ransformer）架构——一个 16 亿参数、完全由 Causal Transformer 块构建的“无 CNN”同构音频 tokenizer。
+
+- **统一离散桥接**：为 MOSS‑TTS、MOSS‑TTSD、MOSS‑VoiceGenerator、MOSS‑SoundEffect 与 MOSS‑TTS‑Realtime 提供共享骨干，使家族内音频表示一致。
+- **极致压缩与高保真**：将 24kHz 原始音频压缩到 12.5Hz 的极低帧率；采用 32 层残差向量量化（RVQ），支持从 0.125kbps 到 4kbps 的可变码率高保真重建。
+- **超大规模通用音频训练**：从零训练，使用 300 万小时多样化数据（语音、音效与音乐），在开源音频 tokenizer 中达到 SOTA 级重建效果。
+- **原生流式设计**：纯 Causal Transformer 架构专为可扩展性与低时延流式推理而设计，支持实时生产流程。
+
+如需了解更多配置、进阶用法与评测指标，请访问 [MOSS-Audio-Tokenizer 仓库](https://github.com/OpenMOSS/MOSS-Audio-Tokenizer)。
+
+<p align="center">
+  <img src="./assets/arch_moss_audio_tokenizer.png" alt="MOSS Audio Tokenizer 架构示意" width="100%" />
+  MOSS Audio Tokenizer 架构图
+</p>
+
+<a id="model-weights"></a>
+### 模型权重
+
+| Model | Hugging Face | ModelScope |
+|:-----:|:------------:|:----------:|
+| **MOSS-Audio-Tokenizer** | [![Hugging Face](https://img.shields.io/badge/Huggingface-Model-orange?logo=huggingface)](https://huggingface.co/OpenMOSS-Team/MOSS-Audio-Tokenizer) | [![ModelScope](https://img.shields.io/badge/ModelScope-Model-lightgrey?logo=modelscope)](https://modelscope.cn/models/openmoss/MOSS-Audio-Tokenizer) |
+
+### 重建质量客观评测
+
+我们在 LibriSpeech test-clean 子集上，对比 **MOSS Audio Tokenizer** 与多个开源音频 tokenizer 的 SIM、STOI、PESQ-NB、PESQ-WB 指标，并通过调节 RVQ 码本数量来控制码率。MOSS Audio Tokenizer 在 0–4 kbps 的比特率上的重建质量领先其他开源音频 tokenizer。
+
+<p align="center">
+  <img src="./assets/evaluation_moss_audio_tokenizer.png" alt="LibriSpeech objective metrics for audio tokenizers" width="90%" />
+</p>
+
+
+## 证书
+
+MOSS-TTS 家族中的模型使用 Apache License 2.0 许可证。
+
+## 引用
+
+```bibtex
+```
+## 星标历史数据
+
+[![Star History Chart](https://api.star-history.com/svg?repos=OpenMOSS/MOSS-TTS&type=date&legend=top-left)](https://www.star-history.com/#OpenMOSS/MOSS-TTS&type=date&legend=top-left)

assets/audio/reference_02_s1.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1816ab428334ba2de49dcb8b0a10e17eb1835f7f1f7bcda13504e88f46bed1e8
+size 249284

assets/audio/reference_02_s2.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3bfaef0dd23b0f382630e516a7a8512bed4a16fb07aa9b9c435d2e6e7b0b9215
+size 1134296

assets/audio/reference_en_1.mp3

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:82e4fad862ccb12a4ac609623fe6c275d167f7dcfc7866ca740f38ab169935c6
+size 213836

assets/audio/reference_en_2.mp3

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:582f8e1e3f3792d7b495e159c29a55bb95c4c46e90725c62807b4b12bf341603
+size 322923

assets/audio/reference_en_3.mp3

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4bd9c6ffb765fda23297fae21725bc174a3092d9687c3606f11d00ae0df9fc1e
+size 107943

assets/audio/reference_zh.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e5112b5e2bef2a727534af85da1e56048a5ab5552de7aa7cbb5f48b0fa4f5eec
+size 448172

assets/audio/reference_zh_0.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e5112b5e2bef2a727534af85da1e56048a5ab5552de7aa7cbb5f48b0fa4f5eec
+size 448172

assets/audio/reference_zh_1.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f4ff19c55d55a37dbbd550e6624a2faf6cfa7fd56a9594456b17fbe3838b2245
+size 1480128

assets/audio/reference_zh_2.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e1686d3e2b1fe2f6b079cf6a41a9cd9ba31c8f9d3cfe03ff411dd0359641c0c8
+size 505586

assets/audio/reference_zh_3.mp3

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cffa7c5d91c28895caf51c38418af9651c82a4e16a8e4c04e10991bf80cc04cc
+size 347949

assets/text/moss_tts_example_texts.jsonl

@@ -0,0 +1,8 @@
+{"id":"zh/0","language":"zh","role":"可爱的小女孩","text":"亲爱的你，\n你好呀。\n今天，我想用最认真、最温柔的声音，对你说一些重要的话。\n这些话，像一颗小小的星星，希望能在你的心里慢慢发光。"}
+{"id":"zh/1","language":"zh","role":"吴俊全老师","text":"从1948年9月12日至1949年1月31日，连续组织了震惊世界的辽沈、淮海、平津三个大战役，这一百四十二个昼夜中，双方统帅部和各级指挥部所拍发的电码讯号错综交汇，织成一面无形的网，从大气层覆盖下来，于是便注定了中国的山川将会怎样排列，流云又当如何变幻。"}
+{"id":"zh/2","language":"zh","role":"原神胡桃","text":"嘿——你在听吗？\n嗯，不说话也没关系啦，反正我已经习惯自言自语了。\n我是胡桃，往生堂第七十七代堂主。\n别紧张别紧张，我今天不是来“请你喝茶”的——至少现在还不是。\n很多人一听到“往生堂”，就皱起眉头，好像我一开口，空气都要凉三分。\n可你看啊，太阳每天都会落下，可谁会因此不喜欢黄昏呢？\n生与死也是一样的道理嘛——\n不是终点，而是换一条路走走。"}
+{"id":"zh/3","language":"zh","role":"明星杨幂","text":"有些人喜欢被照顾，\n而我更习惯照亮自己。\n\n不是不需要依靠，\n只是明白——\n真正能陪你走到最后的，\n从来都不是运气。\n\n我见过凌晨四点的城市，\n也见过掌声散去后的安静。\n那些看起来毫不费力的从容，\n其实都藏着一次次咬牙坚持。"}
+{"id":"en/0","language":"en","role":"Taylor Swift","text":"Tonight, I just want to take a second and breathe this in with you.\nBecause moments like this don’t happen by accident. They’re built—one lyric at a time, one late night at a time, one brave decision at a time. They’re built by people who keep showing up, even when life is loud, even when the world is heavy, even when they’re not sure anyone sees the effort they’re making."}
+{"id":"en/1","language":"en","role":"Iron Man","text":"Look, I know what you’re thinking. Here he goes again. The guy in the metal suit, the walking ego with a repulsor problem, about to make a speech like it’s a press conference and I’m getting paid by the syllable. Relax. This one isn’t for the cameras. No sponsors, no applause, no clever angle that makes me look taller than I already am."}
+{"id":"en/2","language":"en","role":"David Attenborough","text":"In the quiet hours before dawn, the world looks unfinished. Streets are empty, windows are dark, and the air holds its breath as if waiting for a cue. But beneath the stillness, everything is moving. Water is traveling through pipes. Electricity is humming along invisible lines. Seeds are pushing against soil. Somewhere, a hand reaches for a switch, and a day begins."}
+{"id":"en/3","language":"en","role":"Rick Sanchez","text":"Look, you keep staring at the sky like it’s a customer service desk, waiting for the universe to hand you a receipt that says your pain was “worth it.” Newsflash: the cosmos doesn’t do refunds, it does entropy. It does random collisions of atoms that occasionally arrange themselves into a biped with anxiety and a subscription to self-importance."}

assets/text/moss_voice_generator_example_texts.jsonl

@@ -0,0 +1,8 @@
+{"id":"zh/0","language":"zh","instruction":"撕心裂肺，声泪俱下的中年女性","text":"皇上，臣妾做不到啊！皇上，您就杀了臣妾吧！"}
+{"id":"zh/1","language":"zh","instruction":"年轻女性，开头傲慢不屑，发现对方身份后秒怂，疯狂道歉，惊慌失措","text":"你谁啊，关你什么事？啊…王总，您好您好，我不知道是您……"}
+{"id":"zh/2","language":"zh","instruction":"疲惫沙哑的老年声音缓慢抱怨，带有轻微呻吟。","text":"哎呀，我的老腰啊，这年纪大了就是不行了。"}
+{"id":"zh/3","language":"zh","instruction":"粗犷急躁的海盗船长，语速快，语调低沉而充满命令，带着一股不容置疑的霸道。","text":"快点！把那箱金币搬过来！速度快点！别磨磨蹭蹭的！我们必须在涨潮之前离开这里，否则就来不及了！"}
+{"id":"en/0","language":"en","instruction":"Mom scolding kid for breaking a vase, then seeing he cut himself, shifting to concern","text":"How many times have I told you not to run in the house?! You could have…… oh honey, you're bleeding! Let me see your hand…… It's okay, baby."}
+{"id":"en/1","language":"en","instruction":"An elderly female voice, slightly nasal and soft, speaking in a frail, polite British tone, conveying subtle discomfort with gentle hesitation.","text":"Achoo! Oh dear, I do believe I'm catching a cold. This dreadful weather is just too much."}
+{"id":"en/2","language":"en","instruction":"Little girl, innocent and curious, high-pitched and adorable","text":"Mommy, why is the sky blue? And why do birds fly? And why-"}
+{"id":"en/3","language":"en","instruction":"Emotional pop ballad with smooth, melodic delivery, slow tempo with gentle vibrato on sustained notes, conveying hope and vulnerability.","text":"Walking down this empty street tonight, searching for a guiding light, stars above shine oh so bright, everything will be alright"}

benchmark_harness.py

@@ -0,0 +1,310 @@
+#!/usr/bin/env python3
+"""
+Comprehensive benchmark harness for MOSS-TTS realtime optimization.
+
+Captures:
+- TTFB (time to first byte)
+- Real-time factor (RTF)
+- End-to-end latency
+- Peak VRAM and steady-state VRAM
+- Per-stage profiling metrics
+- Audio validation via Whisper API
+"""
+
+import os
+import sys
+import json
+import time
+import subprocess
+import argparse
+from pathlib import Path
+from typing import Optional, Dict, Any, List, Tuple
+import statistics
+
+import requests
+import numpy as np
+
+WHISPER_ENDPOINT = os.getenv("WHISPER_ENDPOINT", "http://100.85.200.54:5092/v1")
+TTS_SERVER_HOST = os.getenv("TTS_SERVER_HOST", "http://localhost:8012")
+
+# Test prompts: short, medium, long
+TEST_PROMPTS = {
+    "short": "The quick brown fox jumps over the lazy dog.",
+    "medium": "Artificial intelligence is transforming how we live, work, and interact. From healthcare to education, AI systems are becoming integral to modern society. Yet, as these systems grow more powerful, the need for safety, transparency, and responsible deployment becomes increasingly critical.",
+    "long": "The vast majority of computational models today operate through neural networks that attempt to approximate the behavior of biological brains. These systems are trained on enormous datasets, learning patterns through iterative optimization. However, the mechanisms by which they achieve their remarkable performance remain largely opaque—a phenomenon often described as the 'black box problem'. This opacity creates challenges in critical applications like healthcare and autonomous systems, where understanding model decisions is essential for safety and liability. Recent advances in explainable AI, mechanistic interpretability, and formal verification aim to address these concerns, but significant open questions remain about whether we can ever fully understand the decision-making processes of deep neural networks.",
+}
+
+class BenchmarkResult:
+    """Encapsulates a single benchmark run."""
+    
+    def __init__(self, prompt_type: str, prompt_text: str):
+        self.prompt_type = prompt_type
+        self.prompt_text = prompt_text
+        self.ttfb_sec = None
+        self.total_latency_sec = None
+        self.audio_duration_sec = None
+        self.rtf = None
+        self.audio_bytes = None
+        self.stage_timings = {}
+        self.response_headers = {}
+        self.whisper_transcript = None
+        self.similarity_score = None
+        self.error = None
+        self.timestamp = time.time()
+    
+    def to_dict(self) -> Dict[str, Any]:
+        return {
+            "prompt_type": self.prompt_type,
+            "prompt_length": len(self.prompt_text),
+            "ttfb_sec": self.ttfb_sec,
+            "total_latency_sec": self.total_latency_sec,
+            "audio_duration_sec": self.audio_duration_sec,
+            "rtf": self.rtf,
+            "audio_bytes": self.audio_bytes,
+            "stage_timings": self.stage_timings,
+            "whisper_transcript": self.whisper_transcript,
+            "similarity_score": self.similarity_score,
+            "error": self.error,
+        }
+
+def extract_stage_timings(headers: Dict[str, str]) -> Dict[str, float]:
+    """Extract profiling timings from response headers."""
+    timings = {}
+    for key, value in headers.items():
+        if key.lower().startswith("x-moss-stage-"):
+            stage_name = key.lower().replace("x-moss-stage-", "").replace("-time-ms", "")
+            try:
+                timings[stage_name] = float(value) / 1000.0  # Convert ms to sec
+            except (ValueError, TypeError):
+                pass
+        elif key.lower().startswith("x-moss-infer-"):
+            infer_name = key.lower().replace("x-moss-infer-", "").replace("-ms", "")
+            try:
+                timings[infer_name] = float(value) / 1000.0
+            except (ValueError, TypeError):
+                pass
+    return timings
+
+def estimate_audio_duration(audio_bytes: int, sample_rate: int = 24000, channels: int = 1, bytes_per_sample: int = 2) -> float:
+    """Estimate audio duration from byte count."""
+    if bytes_per_sample <= 0:
+        return 0.0
+    samples = audio_bytes / (channels * bytes_per_sample)
+    return samples / sample_rate
+
+def transcribe_audio_with_whisper(audio_data: bytes, format_type: str = "pcm") -> Optional[str]:
+    """Send audio to remote Whisper endpoint for transcription."""
+    try:
+        # PCM data needs to be wrapped in WAV format for Whisper
+        if format_type == "pcm":
+            import struct
+            sample_rate = 24000
+            channels = 1
+            wav_data = create_wav_from_pcm(audio_data, sample_rate, channels)
+        else:
+            wav_data = audio_data
+        
+        files = {"file": ("audio.wav", wav_data, "audio/wav")}
+        response = requests.post(
+            f"{WHISPER_ENDPOINT}/audio/transcriptions",
+            files=files,
+            data={"model": "whisper-1"},
+            timeout=60
+        )
+        response.raise_for_status()
+        result = response.json()
+        return result.get("text", "").strip()
+    except Exception as e:
+        print(f"Whisper transcription failed: {e}")
+        return None
+
+def create_wav_from_pcm(pcm_data: bytes, sample_rate: int = 24000, channels: int = 1) -> bytes:
+    """Wrap PCM audio in WAV container."""
+    import struct
+    
+    bytes_per_sample = 2  # 16-bit
+    num_samples = len(pcm_data) // (channels * bytes_per_sample)
+    byte_rate = sample_rate * channels * bytes_per_sample
+    block_align = channels * bytes_per_sample
+    
+    wav_header = b"RIFF"
+    wav_header += struct.pack("<I", 36 + len(pcm_data))
+    wav_header += b"WAVE"
+    wav_header += b"fmt "
+    wav_header += struct.pack("<I", 16)
+    wav_header += struct.pack("<H", 1)  # PCM format
+    wav_header += struct.pack("<H", channels)
+    wav_header += struct.pack("<I", sample_rate)
+    wav_header += struct.pack("<I", byte_rate)
+    wav_header += struct.pack("<H", block_align)
+    wav_header += struct.pack("<H", 16)  # Bits per sample
+    wav_header += b"data"
+    wav_header += struct.pack("<I", len(pcm_data))
+    
+    return wav_header + pcm_data
+
+def similarity_score(text1: str, text2: str) -> float:
+    """Simple word overlap similarity (0-1)."""
+    words1 = set(text1.lower().split())
+    words2 = set(text2.lower().split())
+    if not words1 or not words2:
+        return 0.0
+    intersection = len(words1 & words2)
+    union = len(words1 | words2)
+    return intersection / union if union > 0 else 0.0
+
+def run_benchmark(prompt_type: str, prompt_text: str, validate_audio: bool = False) -> BenchmarkResult:
+    """Run a single benchmark iteration."""
+    result = BenchmarkResult(prompt_type, prompt_text)
+    
+    try:
+        # Stream audio from TTS server
+        start_time = time.time()
+        first_byte_time = None
+        audio_data = b""
+        
+        response = requests.post(
+            f"{TTS_SERVER_HOST}/v1/audio/speech",
+            json={
+                "input": prompt_text,
+                "response_format": "pcm",
+                "stream": True
+            },
+            stream=True,
+            timeout=120
+        )
+        response.raise_for_status()
+        result.response_headers = dict(response.headers)
+        
+        for chunk in response.iter_content(chunk_size=8192):
+            if first_byte_time is None:
+                first_byte_time = time.time()
+            audio_data += chunk
+        
+        end_time = time.time()
+        result.ttfb_sec = first_byte_time - start_time if first_byte_time else 0
+        result.total_latency_sec = end_time - start_time
+        result.audio_bytes = len(audio_data)
+        
+        # Extract stage timings from headers
+        result.stage_timings = extract_stage_timings(result.response_headers)
+        
+        # Estimate audio duration and compute RTF
+        result.audio_duration_sec = estimate_audio_duration(result.audio_bytes)
+        if result.total_latency_sec > 0 and result.audio_duration_sec > 0:
+            result.rtf = result.total_latency_sec / result.audio_duration_sec
+        
+        # Optionally validate with Whisper
+        if validate_audio:
+            result.whisper_transcript = transcribe_audio_with_whisper(audio_data)
+            if result.whisper_transcript:
+                result.similarity_score = similarity_score(prompt_text, result.whisper_transcript)
+        
+    except Exception as e:
+        result.error = str(e)
+        print(f"Error during benchmark: {e}")
+    
+    return result
+
+def run_full_benchmark(validate_audio: bool = False, runs_per_prompt: int = 3) -> Dict[str, Any]:
+    """Run full benchmark suite."""
+    all_results = []
+    summary = {
+        "timestamp": time.time(),
+        "server_url": TTS_SERVER_HOST,
+        "whisper_endpoint": WHISPER_ENDPOINT if validate_audio else "disabled",
+        "results_by_type": {},
+    }
+    
+    for prompt_type, prompt_text in TEST_PROMPTS.items():
+        print(f"\n=== Benchmarking {prompt_type} prompt ({len(prompt_text)} chars) ===")
+        prompt_results = []
+        
+        for i in range(runs_per_prompt):
+            print(f"  Run {i+1}/{runs_per_prompt}...", end=" ", flush=True)
+            result = run_benchmark(prompt_type, prompt_text, validate_audio)
+            prompt_results.append(result)
+            all_results.append(result)
+            
+            if result.error:
+                print(f"ERROR: {result.error}")
+            else:
+                print(f"TTFB={result.ttfb_sec:.3f}s RTF={result.rtf:.2f}x TotalTime={result.total_latency_sec:.3f}s")
+            
+            time.sleep(0.5)  # Brief pause between runs
+        
+        # Summarize this prompt type
+        successful_results = [r for r in prompt_results if not r.error]
+        if successful_results:
+            ttfbs = [r.ttfb_sec for r in successful_results if r.ttfb_sec is not None]
+            rtfs = [r.rtf for r in successful_results if r.rtf is not None]
+            latencies = [r.total_latency_sec for r in successful_results]
+            similarities = [r.similarity_score for r in successful_results if r.similarity_score is not None]
+            
+            prompt_summary = {
+                "runs": len(prompt_results),
+                "successful": len(successful_results),
+                "ttfb": {
+                    "mean_sec": statistics.mean(ttfbs) if ttfbs else None,
+                    "p50_sec": sorted(ttfbs)[len(ttfbs)//2] if ttfbs else None,
+                    "p95_sec": sorted(ttfbs)[int(len(ttfbs)*0.95)] if ttfbs else None,
+                },
+                "rtf": {
+                    "mean": statistics.mean(rtfs) if rtfs else None,
+                    "p50": sorted(rtfs)[len(rtfs)//2] if rtfs else None,
+                    "p95": sorted(rtfs)[int(len(rtfs)*0.95)] if rtfs else None,
+                },
+                "latency": {
+                    "mean_sec": statistics.mean(latencies) if latencies else None,
+                    "min_sec": min(latencies) if latencies else None,
+                    "max_sec": max(latencies) if latencies else None,
+                },
+                "similarity_score": {
+                    "mean": statistics.mean(similarities) if similarities else None,
+                } if similarities else None,
+            }
+            summary["results_by_type"][prompt_type] = prompt_summary
+            
+            # Print stage breakdown for first successful run
+            if successful_results[0].stage_timings:
+                print(f"  Stage breakdown (first run):")
+                for stage, time_sec in sorted(successful_results[0].stage_timings.items()):
+                    print(f"    {stage}: {time_sec*1000:.1f}ms")
+    
+    summary["all_results"] = [r.to_dict() for r in all_results]
+    return summary
+
+def main():
+    parser = argparse.ArgumentParser(description="Benchmark MOSS-TTS realtime optimizations")
+    parser.add_argument("--validate-audio", action="store_true", help="Validate generated audio via Whisper API")
+    parser.add_argument("--runs", type=int, default=3, help="Runs per prompt type")
+    parser.add_argument("--output", type=str, default=None, help="Save results to JSON file")
+    args = parser.parse_args()
+    
+    print("Starting MOSS-TTS Realtime Benchmark Suite")
+    print(f"TTS Server: {TTS_SERVER_HOST}")
+    print(f"Whisper Endpoint: {WHISPER_ENDPOINT if args.validate_audio else 'disabled'}")
+    
+    results = run_full_benchmark(validate_audio=args.validate_audio, runs_per_prompt=args.runs)
+    
+    # Print summary
+    print("\n" + "="*60)
+    print("BENCHMARK SUMMARY")
+    print("="*60)
+    for prompt_type, summary in results["results_by_type"].items():
+        print(f"\n{prompt_type.upper()}:")
+        print(f"  TTFB (mean): {summary['ttfb']['mean_sec']*1000:.1f}ms")
+        print(f"  RTF (mean):  {summary['rtf']['mean']:.2f}x")
+        print(f"  Latency:     {summary['latency']['mean_sec']:.3f}s")
+        if summary.get("similarity_score"):
+            print(f"  Whisper sim: {summary['similarity_score']['mean']:.2%}")
+    
+    if args.output:
+        with open(args.output, "w") as f:
+            json.dump(results, f, indent=2)
+        print(f"\nResults saved to {args.output}")
+    
+    return 0
+
+if __name__ == "__main__":
+    sys.exit(main())

clis/moss_sound_effect_app.py

@@ -0,0 +1,347 @@
+import argparse
+import functools
+import importlib.util
+import time
+
+import gradio as gr
+import numpy as np
+import torch
+from transformers import AutoModel, AutoProcessor
+
+# Disable the broken cuDNN SDPA backend
+torch.backends.cuda.enable_cudnn_sdp(False)
+# Keep these enabled as fallbacks
+torch.backends.cuda.enable_flash_sdp(True)
+torch.backends.cuda.enable_mem_efficient_sdp(True)
+torch.backends.cuda.enable_math_sdp(True)
+
+MODEL_PATH = "OpenMOSS-Team/MOSS-SoundEffect"
+DEFAULT_ATTN_IMPLEMENTATION = "auto"
+DEFAULT_MAX_NEW_TOKENS = 4096
+TOKENS_PER_SECOND = 12.5
+
+
+@functools.lru_cache(maxsize=1)
+def load_backend(model_path: str, device_str: str, attn_implementation: str):
+    device = torch.device(device_str if torch.cuda.is_available() else "cpu")
+    dtype = torch.bfloat16 if device.type == "cuda" else torch.float32
+    resolved_attn_implementation = resolve_attn_implementation(
+        requested=attn_implementation,
+        device=device,
+        dtype=dtype,
+    )
+
+    processor = AutoProcessor.from_pretrained(
+        model_path,
+        trust_remote_code=True,
+    )
+    if hasattr(processor, "audio_tokenizer"):
+        processor.audio_tokenizer = processor.audio_tokenizer.to(device)
+
+    model_kwargs = {
+        "trust_remote_code": True,
+        "torch_dtype": dtype,
+    }
+    if resolved_attn_implementation:
+        model_kwargs["attn_implementation"] = resolved_attn_implementation
+
+    model = AutoModel.from_pretrained(model_path, **model_kwargs).to(device)
+    model.eval()
+
+    sample_rate = int(getattr(processor.model_config, "sampling_rate", 24000))
+    return model, processor, device, sample_rate
+
+
+def resolve_attn_implementation(requested: str, device: torch.device, dtype: torch.dtype) -> str | None:
+    requested_norm = (requested or "").strip().lower()
+
+    if requested_norm in {"none"}:
+        return None
+
+    if requested_norm not in {"", "auto"}:
+        return requested
+
+    # Prefer FlashAttention 2 when package + device conditions are met.
+    if (
+        device.type == "cuda"
+        and importlib.util.find_spec("flash_attn") is not None
+        and dtype in {torch.float16, torch.bfloat16}
+    ):
+        major, _ = torch.cuda.get_device_capability(device)
+        if major >= 8:
+            return "flash_attention_2"
+
+    # CUDA fallback: use PyTorch SDPA kernels.
+    if device.type == "cuda":
+        return "sdpa"
+
+    # CPU fallback.
+    return "eager"
+
+
+def build_conversation(ambient_sound: str, duration_seconds: float, processor):
+    ambient_sound = (ambient_sound or "").strip()
+    if not ambient_sound:
+        raise ValueError("Please enter an ambient sound description.")
+
+    expected_tokens = max(1, int(float(duration_seconds) * TOKENS_PER_SECOND))
+    user_kwargs = {
+        "ambient_sound": ambient_sound,
+        "tokens": expected_tokens,
+    }
+
+    return [[processor.build_user_message(**user_kwargs)]], expected_tokens
+
+
+def run_inference(
+    ambient_sound: str,
+    duration_seconds: float,
+    temperature: float,
+    top_p: float,
+    top_k: int,
+    repetition_penalty: float,
+    max_new_tokens: int,
+    model_path: str,
+    device: str,
+    attn_implementation: str,
+):
+    started_at = time.monotonic()
+    model, processor, torch_device, sample_rate = load_backend(
+        model_path=model_path,
+        device_str=device,
+        attn_implementation=attn_implementation,
+    )
+
+    conversations, expected_tokens = build_conversation(
+        ambient_sound=ambient_sound,
+        duration_seconds=duration_seconds,
+        processor=processor,
+    )
+
+    batch = processor(conversations, mode="generation")
+    input_ids = batch["input_ids"].to(torch_device)
+    attention_mask = batch["attention_mask"].to(torch_device)
+
+    with torch.no_grad():
+        outputs = model.generate(
+            input_ids=input_ids,
+            attention_mask=attention_mask,
+            max_new_tokens=int(max_new_tokens),
+            audio_temperature=float(temperature),
+            audio_top_p=float(top_p),
+            audio_top_k=int(top_k),
+            audio_repetition_penalty=float(repetition_penalty),
+        )
+
+    messages = processor.decode(outputs)
+    if not messages or messages[0] is None:
+        raise RuntimeError("The model did not return a decodable audio result.")
+
+    audio = messages[0].audio_codes_list[0]
+    if isinstance(audio, torch.Tensor):
+        audio_np = audio.detach().float().cpu().numpy()
+    else:
+        audio_np = np.asarray(audio, dtype=np.float32)
+
+    if audio_np.ndim > 1:
+        audio_np = audio_np.reshape(-1)
+    audio_np = audio_np.astype(np.float32, copy=False)
+
+    elapsed = time.monotonic() - started_at
+    status = (
+        f"Done | elapsed: {elapsed:.2f}s | "
+        f"duration_seconds={float(duration_seconds):.0f}, expected_tokens={int(expected_tokens)}, "
+        f"max_new_tokens={int(max_new_tokens)}, "
+        f"audio_temperature={float(temperature):.2f}, audio_top_p={float(top_p):.2f}, "
+        f"audio_top_k={int(top_k)}, audio_repetition_penalty={float(repetition_penalty):.2f}"
+    )
+    return (sample_rate, audio_np), status
+
+
+def build_demo(args: argparse.Namespace):
+    custom_css = """
+    :root {
+      --bg: #f6f7f8;
+      --panel: #ffffff;
+      --ink: #111418;
+      --muted: #4d5562;
+      --line: #e5e7eb;
+      --accent: #0f766e;
+    }
+    .gradio-container {
+      background: linear-gradient(180deg, #f7f8fa 0%, #f3f5f7 100%);
+      color: var(--ink);
+    }
+    .app-card {
+      border: 1px solid var(--line);
+      border-radius: 16px;
+      background: var(--panel);
+      padding: 14px;
+    }
+    .app-title {
+      font-size: 22px;
+      font-weight: 700;
+      margin-bottom: 6px;
+      letter-spacing: 0.2px;
+    }
+    .app-subtitle {
+      color: var(--muted);
+      font-size: 14px;
+      margin-bottom: 8px;
+    }
+    #output_audio {
+      padding-bottom: 12px;
+      margin-bottom: 8px;
+      overflow: hidden !important;
+    }
+    #output_audio > .wrap {
+      overflow: hidden !important;
+    }
+    #output_audio audio {
+      margin-bottom: 6px;
+    }
+    #run-btn {
+      background: var(--accent);
+      border: none;
+    }
+    """
+
+    with gr.Blocks(title="MOSS-SoundEffect Demo", css=custom_css) as demo:
+        gr.Markdown(
+            """
+            <div class="app-card">
+              <div class="app-title">MOSS-SoundEffect</div>
+              <div class="app-subtitle">Generate ambient sounds and sound effects from text descriptions.</div>
+            </div>
+            """
+        )
+
+        with gr.Row(equal_height=False):
+            with gr.Column(scale=3):
+                ambient_sound = gr.Textbox(
+                    label="Ambient Sound Description",
+                    lines=8,
+                    placeholder="Example: Thunder rolls in the distance while heavy rain falls on a metal roof.",
+                )
+                duration_seconds = gr.Slider(
+                    minimum=1,
+                    maximum=60,
+                    step=1,
+                    value=10,
+                    label="Duration (seconds)",
+                )
+
+                with gr.Accordion("Sampling Parameters (Audio)", open=True):
+                    temperature = gr.Slider(
+                        minimum=0.1,
+                        maximum=3.0,
+                        step=0.05,
+                        value=1.5,
+                        label="temperature",
+                    )
+                    top_p = gr.Slider(
+                        minimum=0.1,
+                        maximum=1.0,
+                        step=0.01,
+                        value=0.6,
+                        label="top_p",
+                    )
+                    top_k = gr.Slider(
+                        minimum=1,
+                        maximum=200,
+                        step=1,
+                        value=50,
+                        label="top_k",
+                    )
+                    repetition_penalty = gr.Slider(
+                        minimum=0.8,
+                        maximum=2.0,
+                        step=0.05,
+                        value=1.2,
+                        label="repetition_penalty",
+                    )
+                    max_new_tokens = gr.Slider(
+                        minimum=256,
+                        maximum=8192,
+                        step=128,
+                        value=DEFAULT_MAX_NEW_TOKENS,
+                        label="max_new_tokens",
+                    )
+
+                run_btn = gr.Button("Generate Sound Effect", variant="primary", elem_id="run-btn")
+
+            with gr.Column(scale=2):
+                output_audio = gr.Audio(label="Output Audio", type="numpy", elem_id="output_audio")
+                status = gr.Textbox(label="Status", lines=4, interactive=False)
+
+        run_btn.click(
+            fn=lambda ambient_sound, duration_seconds, temperature, top_p, top_k, repetition_penalty, max_new_tokens: run_inference(
+                ambient_sound=ambient_sound,
+                duration_seconds=duration_seconds,
+                temperature=temperature,
+                top_p=top_p,
+                top_k=top_k,
+                repetition_penalty=repetition_penalty,
+                max_new_tokens=max_new_tokens,
+                model_path=args.model_path,
+                device=args.device,
+                attn_implementation=args.attn_implementation,
+            ),
+            inputs=[
+                ambient_sound,
+                duration_seconds,
+                temperature,
+                top_p,
+                top_k,
+                repetition_penalty,
+                max_new_tokens,
+            ],
+            outputs=[output_audio, status],
+        )
+    return demo
+
+
+def main():
+    parser = argparse.ArgumentParser(description="MOSS-SoundEffect Gradio Demo")
+    parser.add_argument("--model_path", type=str, default=MODEL_PATH)
+    parser.add_argument("--device", type=str, default="cuda:0")
+    parser.add_argument("--attn_implementation", type=str, default=DEFAULT_ATTN_IMPLEMENTATION)
+    parser.add_argument("--host", type=str, default="0.0.0.0")
+    parser.add_argument("--port", type=int, default=7861)
+    parser.add_argument("--share", action="store_true")
+    args = parser.parse_args()
+
+    runtime_device = torch.device(args.device if torch.cuda.is_available() else "cpu")
+    runtime_dtype = torch.bfloat16 if runtime_device.type == "cuda" else torch.float32
+    args.attn_implementation = resolve_attn_implementation(
+        requested=args.attn_implementation,
+        device=runtime_device,
+        dtype=runtime_dtype,
+    ) or "none"
+    print(f"[INFO] Using attn_implementation={args.attn_implementation}", flush=True)
+
+    preload_started_at = time.monotonic()
+    print(
+        f"[Startup] Preloading backend: model={args.model_path}, device={args.device}, attn={args.attn_implementation}",
+        flush=True,
+    )
+    load_backend(
+        model_path=args.model_path,
+        device_str=args.device,
+        attn_implementation=args.attn_implementation,
+    )
+    print(
+        f"[Startup] Backend preload finished in {time.monotonic() - preload_started_at:.2f}s",
+        flush=True,
+    )
+
+    demo = build_demo(args)
+    demo.queue(max_size=16, default_concurrency_limit=1).launch(
+        server_name=args.host,
+        server_port=args.port,
+        share=args.share,
+    )
+
+
+if __name__ == "__main__":
+    main()

clis/moss_tts_app.py

@@ -0,0 +1,621 @@
+import argparse
+import functools
+import importlib.util
+from pathlib import Path
+import re
+import time
+import orjson
+
+import gradio as gr
+import numpy as np
+import torch
+from transformers import AutoModel, AutoProcessor
+
+# Disable the broken cuDNN SDPA backend
+torch.backends.cuda.enable_cudnn_sdp(False)
+# Keep these enabled as fallbacks
+torch.backends.cuda.enable_flash_sdp(True)
+torch.backends.cuda.enable_mem_efficient_sdp(True)
+torch.backends.cuda.enable_math_sdp(True)
+
+MODEL_PATH = "OpenMOSS-Team/MOSS-TTS"
+DEFAULT_ATTN_IMPLEMENTATION = "auto"
+DEFAULT_MAX_NEW_TOKENS = 4096
+CONTINUATION_NOTICE = (
+    "Continuation mode is active. Make sure the reference audio transcript is prepended to the input text."
+)
+
+MODE_CLONE = "Clone"
+MODE_CONTINUE = "Continuation"
+MODE_CONTINUE_CLONE = "Continuation + Clone"
+ZH_TOKENS_PER_CHAR = 3.098411951313033
+EN_TOKENS_PER_CHAR = 0.8673376262755219
+REFERENCE_AUDIO_DIR = Path(__file__).resolve().parent.parent / "assets" / "audio"
+EXAMPLE_TEXTS_JSONL_PATH = Path(__file__).resolve().parent.parent / "assets" / "text" / "moss_tts_example_texts.jsonl"
+
+
+def _parse_example_id(example_id: str) -> tuple[str, int] | None:
+    matched = re.fullmatch(r"(zh|en)/(\d+)", (example_id or "").strip())
+    if matched is None:
+        return None
+    return matched.group(1), int(matched.group(2))
+
+
+def _resolve_reference_audio_path(language: str, index: int) -> Path | None:
+    stem_candidates = [f"reference_{language}_{index}"]
+    for stem in stem_candidates:
+        for ext in (".wav", ".mp3"):
+            audio_path = REFERENCE_AUDIO_DIR / f"{stem}{ext}"
+            if audio_path.exists():
+                return audio_path
+    return None
+
+
+def build_example_rows() -> list[tuple[str, str, str]]:
+    rows: list[tuple[str, str, str]] = []
+
+    with open(EXAMPLE_TEXTS_JSONL_PATH, "rb") as f:
+        for line in f:
+            if not line.strip():
+                continue
+            sample = orjson.loads(line)
+            parsed = _parse_example_id(sample.get("id", ""))
+            if parsed is None:
+                continue
+
+            language, index = parsed
+            text = str(sample.get("text", "")).strip()
+            audio_path = _resolve_reference_audio_path(language, index)
+            if audio_path is None:
+                continue
+
+            rows.append((sample['role'], str(audio_path), text))
+
+    return rows
+
+
+EXAMPLE_ROWS = build_example_rows()
+
+
+@functools.lru_cache(maxsize=1)
+def load_backend(model_path: str, device_str: str, attn_implementation: str):
+    device = torch.device(device_str if torch.cuda.is_available() else "cpu")
+    dtype = torch.bfloat16 if device.type == "cuda" else torch.float32
+    resolved_attn_implementation = resolve_attn_implementation(
+        requested=attn_implementation,
+        device=device,
+        dtype=dtype,
+    )
+
+    processor = AutoProcessor.from_pretrained(
+        model_path,
+        trust_remote_code=True,
+    )
+    if hasattr(processor, "audio_tokenizer"):
+        processor.audio_tokenizer = processor.audio_tokenizer.to(device)
+
+    model_kwargs = {
+        "trust_remote_code": True,
+        "torch_dtype": dtype,
+    }
+    if resolved_attn_implementation:
+        model_kwargs["attn_implementation"] = resolved_attn_implementation
+
+    model = AutoModel.from_pretrained(model_path, **model_kwargs).to(device)
+    model.eval()
+
+    sample_rate = int(getattr(processor.model_config, "sampling_rate", 24000))
+    return model, processor, device, sample_rate
+
+
+def resolve_attn_implementation(requested: str, device: torch.device, dtype: torch.dtype) -> str | None:
+    requested_norm = (requested or "").strip().lower()
+
+    if requested_norm in {"none"}:
+        return None
+
+    if requested_norm not in {"", "auto"}:
+        return requested
+
+    # Prefer FlashAttention 2 when package + device conditions are met.
+    if (
+        device.type == "cuda"
+        and importlib.util.find_spec("flash_attn") is not None
+        and dtype in {torch.float16, torch.bfloat16}
+    ):
+        major, _ = torch.cuda.get_device_capability(device)
+        if major >= 8:
+            return "flash_attention_2"
+
+    # CUDA fallback: use PyTorch SDPA kernels.
+    if device.type == "cuda":
+        return "sdpa"
+
+    # CPU fallback.
+    return "eager"
+
+
+def detect_text_language(text: str) -> str:
+    zh_chars = len(re.findall(r"[\u4e00-\u9fff]", text))
+    en_chars = len(re.findall(r"[A-Za-z]", text))
+    if zh_chars == 0 and en_chars == 0:
+        return "en"
+    return "zh" if zh_chars >= en_chars else "en"
+
+
+def supports_duration_control(mode_with_reference: str) -> bool:
+    return mode_with_reference not in {MODE_CONTINUE, MODE_CONTINUE_CLONE}
+
+
+def estimate_duration_tokens(text: str) -> tuple[str, int, int, int]:
+    normalized = text or ""
+    effective_len = max(len(normalized), 1)
+    language = detect_text_language(normalized)
+    factor = ZH_TOKENS_PER_CHAR if language == "zh" else EN_TOKENS_PER_CHAR
+    default_tokens = max(1, int(effective_len * factor))
+    min_tokens = max(1, int(default_tokens * 0.5))
+    max_tokens = max(min_tokens, int(default_tokens * 1.5))
+    return language, default_tokens, min_tokens, max_tokens
+
+
+def update_duration_controls(
+    enabled: bool,
+    text: str,
+    current_tokens: float | int | None,
+    mode_with_reference: str,
+):
+    if not supports_duration_control(mode_with_reference):
+        return (
+            gr.update(visible=False),
+            "Duration control is disabled for Continuation modes.",
+            gr.update(value=False, interactive=False),
+        )
+
+    checkbox_update = gr.update(interactive=True)
+    if not enabled:
+        return gr.update(visible=False), "Duration control is disabled.", checkbox_update
+
+    language, default_tokens, min_tokens, max_tokens = estimate_duration_tokens(text)
+    # Slider is initialized with value=1 as a placeholder; treat it as "unset"
+    # so first-time estimation uses the computed default instead of clamping to min.
+    if current_tokens is None or int(current_tokens) == 1:
+        slider_value = default_tokens
+    else:
+        slider_value = int(current_tokens)
+        slider_value = max(min_tokens, min(max_tokens, slider_value))
+
+    language_label = "Chinese" if language == "zh" else "English"
+    hint = (
+        f"Duration control enabled | detected language: {language_label} | "
+        f"default={default_tokens}, range=[{min_tokens}, {max_tokens}]"
+    )
+    return (
+        gr.update(
+            visible=True,
+            minimum=min_tokens,
+            maximum=max_tokens,
+            value=slider_value,
+            step=1,
+        ),
+        hint,
+        checkbox_update,
+    )
+
+
+def build_conversation(
+    text: str,
+    reference_audio: str | None,
+    mode_with_reference: str,
+    expected_tokens: int | None,
+    processor,
+):
+    text = (text or "").strip()
+    if not text:
+        raise ValueError("Please enter text to synthesize.")
+
+    user_kwargs = {"text": text}
+    if expected_tokens is not None:
+        user_kwargs["tokens"] = int(expected_tokens)
+
+    if not reference_audio:
+        conversations = [[processor.build_user_message(**user_kwargs)]]
+        return conversations, "generation", "Direct Generation"
+
+    if mode_with_reference == MODE_CLONE:
+        clone_kwargs = dict(user_kwargs)
+        clone_kwargs["reference"] = [reference_audio]
+        conversations = [[processor.build_user_message(**clone_kwargs)]]
+        return conversations, "generation", MODE_CLONE
+
+    if mode_with_reference == MODE_CONTINUE:
+        conversations = [
+            [
+                processor.build_user_message(**user_kwargs),
+                processor.build_assistant_message(audio_codes_list=[reference_audio]),
+            ]
+        ]
+        return conversations, "continuation", MODE_CONTINUE
+
+    continue_clone_kwargs = dict(user_kwargs)
+    continue_clone_kwargs["reference"] = [reference_audio]
+    conversations = [
+        [
+            processor.build_user_message(**continue_clone_kwargs),
+            processor.build_assistant_message(audio_codes_list=[reference_audio]),
+        ]
+    ]
+    return conversations, "continuation", MODE_CONTINUE_CLONE
+
+
+def render_mode_hint(reference_audio: str | None, mode_with_reference: str):
+    if not reference_audio:
+        return "Current mode: **Direct Generation** (no reference audio uploaded)"
+    if mode_with_reference == MODE_CLONE:
+        return "Current mode: **Clone** (speaker timbre will be cloned from the reference audio)"
+    return f"Current mode: **{mode_with_reference}**  \n> {CONTINUATION_NOTICE}"
+
+
+def apply_example_selection(
+    mode_with_reference: str,
+    duration_control_enabled: bool,
+    duration_tokens: int,
+    evt: gr.SelectData,
+):
+    if evt is None or evt.index is None:
+        return gr.update(), gr.update(), gr.update(), gr.update(), gr.update(), gr.update()
+
+    if isinstance(evt.index, (tuple, list)):
+        row_idx = int(evt.index[0])
+    else:
+        row_idx = int(evt.index)
+
+    if row_idx < 0 or row_idx >= len(EXAMPLE_ROWS):
+        return gr.update(), gr.update(), gr.update(), gr.update(), gr.update(), gr.update()
+
+    _, audio_path, example_text = EXAMPLE_ROWS[row_idx]
+    duration_slider_update, duration_hint, duration_checkbox_update = update_duration_controls(
+        duration_control_enabled,
+        example_text,
+        duration_tokens,
+        mode_with_reference,
+    )
+    return (
+        audio_path,
+        example_text,
+        render_mode_hint(audio_path, mode_with_reference),
+        duration_slider_update,
+        duration_hint,
+        duration_checkbox_update,
+    )
+
+
+def run_inference(
+    text: str,
+    reference_audio: str | None,
+    mode_with_reference: str,
+    duration_control_enabled: bool,
+    duration_tokens: int,
+    temperature: float,
+    top_p: float,
+    top_k: int,
+    repetition_penalty: float,
+    model_path: str,
+    device: str,
+    attn_implementation: str,
+    max_new_tokens: int,
+):
+    started_at = time.monotonic()
+    model, processor, torch_device, sample_rate = load_backend(
+        model_path=model_path,
+        device_str=device,
+        attn_implementation=attn_implementation,
+    )
+    duration_enabled = bool(duration_control_enabled and supports_duration_control(mode_with_reference))
+    expected_tokens = int(duration_tokens) if duration_enabled else None
+    conversations, mode, mode_name = build_conversation(
+        text=text,
+        reference_audio=reference_audio,
+        mode_with_reference=mode_with_reference,
+        expected_tokens=expected_tokens,
+        processor=processor,
+    )
+
+    batch = processor(conversations, mode=mode)
+    input_ids = batch["input_ids"].to(torch_device)
+    attention_mask = batch["attention_mask"].to(torch_device)
+
+    with torch.no_grad():
+        outputs = model.generate(
+            input_ids=input_ids,
+            attention_mask=attention_mask,
+            max_new_tokens=int(max_new_tokens),
+            audio_temperature=float(temperature),
+            audio_top_p=float(top_p),
+            audio_top_k=int(top_k),
+            audio_repetition_penalty=float(repetition_penalty),
+        )
+
+    messages = processor.decode(outputs)
+    if not messages or messages[0] is None:
+        raise RuntimeError("The model did not return a decodable audio result.")
+
+    audio = messages[0].audio_codes_list[0]
+    if isinstance(audio, torch.Tensor):
+        audio_np = audio.detach().float().cpu().numpy()
+    else:
+        audio_np = np.asarray(audio, dtype=np.float32)
+
+    if audio_np.ndim > 1:
+        audio_np = audio_np.reshape(-1)
+    audio_np = audio_np.astype(np.float32, copy=False)
+
+    elapsed = time.monotonic() - started_at
+    status = (
+        f"Done | mode: {mode_name} | elapsed: {elapsed:.2f}s | "
+        f"max_new_tokens={int(max_new_tokens)}, "
+        f"expected_tokens={expected_tokens if expected_tokens is not None else 'off'}, "
+        f"audio_temperature={float(temperature):.2f}, audio_top_p={float(top_p):.2f}, "
+        f"audio_top_k={int(top_k)}, audio_repetition_penalty={float(repetition_penalty):.2f}"
+    )
+    return (sample_rate, audio_np), status
+
+
+def build_demo(args: argparse.Namespace):
+    custom_css = """
+    :root {
+      --bg: #f6f7f8;
+      --panel: #ffffff;
+      --ink: #111418;
+      --muted: #4d5562;
+      --line: #e5e7eb;
+      --accent: #0f766e;
+    }
+    .gradio-container {
+      background: linear-gradient(180deg, #f7f8fa 0%, #f3f5f7 100%);
+      color: var(--ink);
+    }
+    .app-card {
+      border: 1px solid var(--line);
+      border-radius: 16px;
+      background: var(--panel);
+      padding: 14px;
+    }
+    .app-title {
+      font-size: 22px;
+      font-weight: 700;
+      margin-bottom: 6px;
+      letter-spacing: 0.2px;
+    }
+    .app-subtitle {
+      color: var(--muted);
+      font-size: 14px;
+      margin-bottom: 8px;
+    }
+    #output_audio {
+      padding-bottom: 12px;
+      margin-bottom: 8px;
+      overflow: hidden !important;
+    }
+    #output_audio > .wrap {
+      overflow: hidden !important;
+    }
+    #output_audio audio {
+      margin-bottom: 6px;
+    }
+    #run-btn {
+      background: var(--accent);
+      border: none;
+    }
+    """
+
+    with gr.Blocks(title="MOSS-TTS Demo", css=custom_css) as demo:
+        gr.Markdown(
+            """
+            <div class="app-card">
+              <div class="app-title">MOSS-TTS</div>
+              <div class="app-subtitle">Minimal UI: Direct Generation, Clone, Continuation, Continuation + Clone</div>
+            </div>
+            """
+        )
+
+        with gr.Row(equal_height=False):
+            with gr.Column(scale=3):
+                text = gr.Textbox(
+                    label="Text",
+                    lines=9,
+                    placeholder="Enter text to synthesize. In continuation modes, prepend the reference audio transcript.",
+                )
+                reference_audio = gr.Audio(
+                    label="Reference Audio (Optional)",
+                    type="filepath",
+                )
+                mode_with_reference = gr.Radio(
+                    choices=[MODE_CLONE, MODE_CONTINUE, MODE_CONTINUE_CLONE],
+                    value=MODE_CLONE,
+                    label="Mode with Reference Audio",
+                    info="If no reference audio is uploaded, Direct Generation will be used automatically.",
+                )
+                mode_hint = gr.Markdown(render_mode_hint(None, MODE_CLONE))
+                duration_control_enabled = gr.Checkbox(
+                    value=False,
+                    label="Enable Duration Control (Expected Audio Tokens)",
+                )
+                duration_tokens = gr.Slider(
+                    minimum=1,
+                    maximum=1,
+                    step=1,
+                    value=1,
+                    label="expected_tokens",
+                    visible=False,
+                )
+                duration_hint = gr.Markdown("Duration control is disabled.")
+
+                with gr.Accordion("Sampling Parameters (Audio)", open=True):
+                    temperature = gr.Slider(
+                        minimum=0.1,
+                        maximum=3.0,
+                        step=0.05,
+                        value=1.7,
+                        label="temperature",
+                    )
+                    top_p = gr.Slider(
+                        minimum=0.1,
+                        maximum=1.0,
+                        step=0.01,
+                        value=0.8,
+                        label="top_p",
+                    )
+                    top_k = gr.Slider(
+                        minimum=1,
+                        maximum=200,
+                        step=1,
+                        value=25,
+                        label="top_k",
+                    )
+                    repetition_penalty = gr.Slider(
+                        minimum=0.8,
+                        maximum=2.0,
+                        step=0.05,
+                        value=1.0,
+                        label="repetition_penalty",
+                    )
+                    max_new_tokens = gr.Slider(
+                        minimum=256,
+                        maximum=8192,
+                        step=128,
+                        value=DEFAULT_MAX_NEW_TOKENS,
+                        label="max_new_tokens",
+                    )
+
+                run_btn = gr.Button("Generate Speech", variant="primary", elem_id="run-btn")
+
+            with gr.Column(scale=2):
+                output_audio = gr.Audio(label="Output Audio", type="numpy", elem_id="output_audio")
+                status = gr.Textbox(label="Status", lines=4, interactive=False)
+                examples_table = gr.Dataframe(
+                    headers=["Reference Speech", "Example Text"],
+                    value=[[name, text] for name, _, text in EXAMPLE_ROWS],
+                    datatype=["str", "str"],
+                    row_count=(len(EXAMPLE_ROWS), "fixed"),
+                    col_count=(2, "fixed"),
+                    interactive=False,
+                    wrap=True,
+                    label="Examples (click a row to fill inputs)",
+                )
+
+        reference_audio.change(
+            fn=render_mode_hint,
+            inputs=[reference_audio, mode_with_reference],
+            outputs=[mode_hint],
+        )
+        mode_with_reference.change(
+            fn=render_mode_hint,
+            inputs=[reference_audio, mode_with_reference],
+            outputs=[mode_hint],
+        )
+        duration_control_enabled.change(
+            fn=update_duration_controls,
+            inputs=[duration_control_enabled, text, duration_tokens, mode_with_reference],
+            outputs=[duration_tokens, duration_hint, duration_control_enabled],
+        )
+        text.change(
+            fn=update_duration_controls,
+            inputs=[duration_control_enabled, text, duration_tokens, mode_with_reference],
+            outputs=[duration_tokens, duration_hint, duration_control_enabled],
+        )
+        mode_with_reference.change(
+            fn=update_duration_controls,
+            inputs=[duration_control_enabled, text, duration_tokens, mode_with_reference],
+            outputs=[duration_tokens, duration_hint, duration_control_enabled],
+        )
+        examples_table.select(
+            fn=apply_example_selection,
+            inputs=[mode_with_reference, duration_control_enabled, duration_tokens],
+            outputs=[
+                reference_audio,
+                text,
+                mode_hint,
+                duration_tokens,
+                duration_hint,
+                duration_control_enabled,
+            ],
+        )
+
+        run_btn.click(
+            fn=lambda text, reference_audio, mode_with_reference, duration_control_enabled, duration_tokens, temperature, top_p, top_k, repetition_penalty, max_new_tokens: run_inference(
+                text=text,
+                reference_audio=reference_audio,
+                mode_with_reference=mode_with_reference,
+                duration_control_enabled=duration_control_enabled,
+                duration_tokens=duration_tokens,
+                temperature=temperature,
+                top_p=top_p,
+                top_k=top_k,
+                repetition_penalty=repetition_penalty,
+                model_path=args.model_path,
+                device=args.device,
+                attn_implementation=args.attn_implementation,
+                max_new_tokens=max_new_tokens,
+            ),
+            inputs=[
+                text,
+                reference_audio,
+                mode_with_reference,
+                duration_control_enabled,
+                duration_tokens,
+                temperature,
+                top_p,
+                top_k,
+                repetition_penalty,
+                max_new_tokens,
+            ],
+            outputs=[output_audio, status],
+        )
+    return demo
+
+
+def main():
+    parser = argparse.ArgumentParser(description="MossTTS Gradio Demo")
+    parser.add_argument("--model_path", type=str, default=MODEL_PATH)
+    parser.add_argument("--device", type=str, default="cuda:0")
+    parser.add_argument("--attn_implementation", type=str, default=DEFAULT_ATTN_IMPLEMENTATION)
+    parser.add_argument("--host", type=str, default="0.0.0.0")
+    parser.add_argument("--port", type=int, default=7860)
+    parser.add_argument("--share", action="store_true")
+    args = parser.parse_args()
+
+    runtime_device = torch.device(args.device if torch.cuda.is_available() else "cpu")
+    runtime_dtype = torch.bfloat16 if runtime_device.type == "cuda" else torch.float32
+    args.attn_implementation = resolve_attn_implementation(
+        requested=args.attn_implementation,
+        device=runtime_device,
+        dtype=runtime_dtype,
+    ) or "none"
+    print(f"[INFO] Using attn_implementation={args.attn_implementation}", flush=True)
+
+    # Preload model/processor at startup to avoid first-request cold start latency.
+    preload_started_at = time.monotonic()
+    print(
+        f"[Startup] Preloading backend: model={args.model_path}, device={args.device}, attn={args.attn_implementation}",
+        flush=True,
+    )
+    load_backend(
+        model_path=args.model_path,
+        device_str=args.device,
+        attn_implementation=args.attn_implementation,
+    )
+    print(
+        f"[Startup] Backend preload finished in {time.monotonic() - preload_started_at:.2f}s",
+        flush=True,
+    )
+
+    demo = build_demo(args)
+    demo.queue(max_size=16, default_concurrency_limit=1).launch(
+        server_name=args.host,
+        server_port=args.port,
+        share=args.share,
+    )
+
+
+if __name__ == "__main__":
+    main()

clis/moss_ttsd_app.py

@@ -0,0 +1,811 @@
+import argparse
+import functools
+import importlib.util
+import re
+import time
+from pathlib import Path
+from typing import Optional
+
+import gradio as gr
+import numpy as np
+import torch
+import torchaudio
+from transformers import AutoModel, AutoProcessor
+
+# Disable the broken cuDNN SDPA backend
+torch.backends.cuda.enable_cudnn_sdp(False)
+# Keep these enabled as fallbacks
+torch.backends.cuda.enable_flash_sdp(True)
+torch.backends.cuda.enable_mem_efficient_sdp(True)
+torch.backends.cuda.enable_math_sdp(True)
+
+MODEL_PATH = "OpenMOSS-Team/MOSS-TTSD-v1.0"
+CODEC_MODEL_PATH = "OpenMOSS-Team/MOSS-Audio-Tokenizer"
+DEFAULT_ATTN_IMPLEMENTATION = "auto"
+DEFAULT_MAX_NEW_TOKENS = 2000
+MIN_SPEAKERS = 1
+MAX_SPEAKERS = 5
+PRESET_REF_AUDIO_S1 = "assets/audio/reference_02_s1.wav"
+PRESET_REF_AUDIO_S2 = "assets/audio/reference_02_s2.wav"
+PRESET_PROMPT_TEXT_S1 = (
+    "[S1] In short, we embarked on a mission to make America great again for all Americans."
+)
+PRESET_PROMPT_TEXT_S2 = (
+    "[S2] NVIDIA reinvented computing for the first time after 60 years. In fact, Erwin at IBM knows quite "
+    "well that the computer has largely been the same since the 60s."
+)
+PRESET_DIALOGUE_TEXT = (
+    "[S1] Listen, let's talk business. China. I'm hearing things.\n"
+    "People are saying they're catching up. Fast. What's the real scoop?\n"
+    "Their AI, is it a threat?\n"
+    "[S2] Well, the pace of innovation there is extraordinary, honestly.\n"
+    "They have the researchers, and they have the drive.\n"
+    "[S1] Extraordinary? I don't like that. I want us to be extraordinary.\n"
+    "Are they winning?\n"
+    "[S2] I wouldn't say winning, but their progress is very promising.\n"
+    "They are building massive clusters. They're very determined.\n"
+    "[S1] Promising. There it is. I hate that word.\n"
+    "When China is promising, it means we're losing.\n"
+    "It's a disaster, Jensen. A total disaster."
+)
+PRESET_EXAMPLES = [
+    {
+        "name": "Quick Start | reference_02_s1/s2",
+        "speaker_count": 2,
+        "s1_audio": PRESET_REF_AUDIO_S1,
+        "s1_prompt": PRESET_PROMPT_TEXT_S1,
+        "s2_audio": PRESET_REF_AUDIO_S2,
+        "s2_prompt": PRESET_PROMPT_TEXT_S2,
+        "dialogue_text": PRESET_DIALOGUE_TEXT,
+    }
+]
+PRESET_DISPLAY_FIELDS = [
+    ("Speaker Count", "speaker_count"),
+    ("S1 Reference Audio (Optional)", "s1_audio"),
+    ("S1 Prompt Text (Required with reference audio)", "s1_prompt"),
+    ("S2 Reference Audio (Optional)", "s2_audio"),
+    ("S2 Prompt Text (Required with reference audio)", "s2_prompt"),
+    ("Dialogue Text", "dialogue_text"),
+]
+
+
+def _build_preset_table_rows():
+    rows = []
+    row_to_preset = []
+    for preset_idx, preset in enumerate(PRESET_EXAMPLES):
+        for field_name, field_key in PRESET_DISPLAY_FIELDS:
+            value = str(preset.get(field_key, ""))
+            if field_key == "dialogue_text":
+                value = value.replace("\n", " ").strip()
+                if len(value) > 120:
+                    value = value[:120] + " ..."
+            rows.append([field_name, value])
+            row_to_preset.append(preset_idx)
+    return rows, row_to_preset
+
+
+PRESET_TABLE_ROWS, PRESET_TABLE_ROW_TO_PRESET = _build_preset_table_rows()
+
+
+def resolve_attn_implementation(requested: str, device: torch.device, dtype: torch.dtype) -> str | None:
+    requested_norm = (requested or "").strip().lower()
+
+    if requested_norm in {"none"}:
+        return None
+
+    if requested_norm not in {"", "auto"}:
+        return requested
+
+    # Prefer FlashAttention 2 when package + device conditions are met.
+    if (
+        device.type == "cuda"
+        and importlib.util.find_spec("flash_attn") is not None
+        and dtype in {torch.float16, torch.bfloat16}
+    ):
+        major, _ = torch.cuda.get_device_capability(device)
+        if major >= 8:
+            return "flash_attention_2"
+
+    # CUDA fallback: use PyTorch SDPA kernels.
+    if device.type == "cuda":
+        return "sdpa"
+
+    # CPU fallback.
+    return "eager"
+
+
+@functools.lru_cache(maxsize=1)
+def load_backend(model_path: str, codec_path: str, device_str: str, attn_implementation: str):
+    device = torch.device(device_str if torch.cuda.is_available() else "cpu")
+    dtype = torch.bfloat16 if device.type == "cuda" else torch.float32
+    resolved_attn_implementation = resolve_attn_implementation(
+        requested=attn_implementation,
+        device=device,
+        dtype=dtype,
+    )
+
+    processor = AutoProcessor.from_pretrained(
+        model_path,
+        trust_remote_code=True,
+        codec_path=codec_path,
+    )
+    if hasattr(processor, "audio_tokenizer"):
+        processor.audio_tokenizer = processor.audio_tokenizer.to(device)
+        processor.audio_tokenizer.eval()
+
+    model_kwargs = {
+        "trust_remote_code": True,
+        "torch_dtype": dtype,
+    }
+    if resolved_attn_implementation:
+        model_kwargs["attn_implementation"] = resolved_attn_implementation
+
+    model = AutoModel.from_pretrained(model_path, **model_kwargs).to(device)
+    model.eval()
+
+    sample_rate = int(getattr(processor.model_config, "sampling_rate", 24000))
+    return model, processor, device, sample_rate
+
+
+def _resample_wav(wav: torch.Tensor, orig_sr: int, target_sr: int) -> torch.Tensor:
+    if int(orig_sr) == int(target_sr):
+        return wav
+    new_num_samples = int(round(wav.shape[-1] * float(target_sr) / float(orig_sr)))
+    if new_num_samples <= 0:
+        raise ValueError(f"Invalid resample length from {orig_sr}Hz to {target_sr}Hz.")
+    return torch.nn.functional.interpolate(
+        wav.unsqueeze(0),
+        size=new_num_samples,
+        mode="linear",
+        align_corners=False,
+    ).squeeze(0)
+
+
+def _load_audio(audio_path: str) -> tuple[torch.Tensor, int]:
+    path = Path(audio_path).expanduser()
+    if not path.exists():
+        raise FileNotFoundError(f"Reference audio not found: {path}")
+
+    wav, sr = torchaudio.load(str(path))
+    if wav.numel() == 0:
+        raise ValueError(f"Reference audio is empty: {path}")
+
+    if wav.shape[0] > 1:
+        wav = wav.mean(dim=0, keepdim=True)
+
+    return wav, int(sr)
+
+
+def normalize_text(text: str) -> str:
+    text = re.sub(r"\[(\d+)\]", r"[S\1]", text)
+    remove_chars = "【】《》（）『』「」" '"-_“”～~‘’'
+
+    segments = re.split(r"(?=\[S\d+\])", text.replace("\n", " "))
+    processed_parts = []
+    for seg in segments:
+        seg = seg.strip()
+        if not seg:
+            continue
+
+        matched = re.match(r"^(\[S\d+\])\s*(.*)", seg)
+        tag, content = matched.groups() if matched else ("", seg)
+
+        content = re.sub(f"[{re.escape(remove_chars)}]", "", content)
+        content = re.sub(r"哈{2,}", "[笑]", content)
+        content = re.sub(r"\b(ha(\s*ha)+)\b", "[laugh]", content, flags=re.IGNORECASE)
+
+        content = content.replace("——", "，")
+        content = content.replace("……", "，")
+        content = content.replace("...", "，")
+        content = content.replace("⸺", "，")
+        content = content.replace("―", "，")
+        content = content.replace("—", "，")
+        content = content.replace("…", "，")
+
+        internal_punct_map = str.maketrans(
+            {"；": "，", ";": ",", "：": "，", ":": ",", "、": "，"}
+        )
+        content = content.translate(internal_punct_map)
+        content = content.strip()
+        content = re.sub(r"([，。？！,.?!])[，。？！,.?!]+", r"\1", content)
+
+        if len(content) > 1:
+            last_ch = "。" if content[-1] == "，" else ("." if content[-1] == "," else content[-1])
+            body = content[:-1].replace("。", "，")
+            content = body + last_ch
+
+        processed_parts.append({"tag": tag, "content": content})
+
+    if not processed_parts:
+        return ""
+
+    merged_lines = []
+    current_tag = processed_parts[0]["tag"]
+    current_content = [processed_parts[0]["content"]]
+    for part in processed_parts[1:]:
+        if part["tag"] == current_tag and current_tag:
+            current_content.append(part["content"])
+        else:
+            merged_lines.append(f"{current_tag}{''.join(current_content)}".strip())
+            current_tag = part["tag"]
+            current_content = [part["content"]]
+    merged_lines.append(f"{current_tag}{''.join(current_content)}".strip())
+
+    return "".join(merged_lines).replace("‘", "'").replace("’", "'")
+
+
+def _validate_dialogue_text(dialogue_text: str, speaker_count: int) -> str:
+    text = (dialogue_text or "").strip()
+    if not text:
+        raise ValueError("Please enter dialogue text.")
+
+    tags = re.findall(r"\[S(\d+)\]", text)
+    if not tags:
+        raise ValueError("Dialogue must include speaker tags like [S1], [S2], ...")
+
+    max_tag = max(int(t) for t in tags)
+    if max_tag > speaker_count:
+        raise ValueError(
+            f"Dialogue contains [S{max_tag}], but speaker count is set to {speaker_count}."
+        )
+    return text
+
+
+def update_speaker_panels(speaker_count: int):
+    count = int(speaker_count)
+    count = max(MIN_SPEAKERS, min(MAX_SPEAKERS, count))
+    return [gr.update(visible=(idx < count)) for idx in range(MAX_SPEAKERS)]
+
+
+def apply_preset_selection(evt: gr.SelectData):
+    if evt is None or evt.index is None:
+        return (
+            gr.update(),
+            gr.update(),
+            gr.update(),
+            gr.update(),
+            gr.update(),
+            gr.update(),
+            *[gr.update() for _ in range(MAX_SPEAKERS)],
+        )
+
+    if isinstance(evt.index, (tuple, list)):
+        row_idx = int(evt.index[0])
+    else:
+        row_idx = int(evt.index)
+
+    if row_idx < 0 or row_idx >= len(PRESET_TABLE_ROW_TO_PRESET):
+        return (
+            gr.update(),
+            gr.update(),
+            gr.update(),
+            gr.update(),
+            gr.update(),
+            gr.update(),
+            *[gr.update() for _ in range(MAX_SPEAKERS)],
+        )
+
+    preset_idx = PRESET_TABLE_ROW_TO_PRESET[row_idx]
+    if preset_idx < 0 or preset_idx >= len(PRESET_EXAMPLES):
+        return (
+            gr.update(),
+            gr.update(),
+            gr.update(),
+            gr.update(),
+            gr.update(),
+            gr.update(),
+            *[gr.update() for _ in range(MAX_SPEAKERS)],
+        )
+
+    preset = PRESET_EXAMPLES[preset_idx]
+    panel_updates = update_speaker_panels(int(preset["speaker_count"]))
+    return (
+        gr.update(value=int(preset["speaker_count"])),
+        gr.update(value=str(preset["s1_audio"])),
+        gr.update(value=str(preset["s1_prompt"])),
+        gr.update(value=str(preset["s2_audio"])),
+        gr.update(value=str(preset["s2_prompt"])),
+        gr.update(value=str(preset["dialogue_text"])),
+        *panel_updates,
+    )
+
+
+def _merge_consecutive_speaker_tags(text: str) -> str:
+    segments = re.split(r"(?=\[S\d+\])", text)
+    if not segments:
+        return text
+
+    merged_parts = []
+    current_tag = None
+    for seg in segments:
+        seg = seg.strip()
+        if not seg:
+            continue
+        matched = re.match(r"^(\[S\d+\])\s*(.*)", seg, re.DOTALL)
+        if not matched:
+            merged_parts.append(seg)
+            continue
+        tag, content = matched.groups()
+        if tag == current_tag:
+            merged_parts.append(content)
+        else:
+            current_tag = tag
+            merged_parts.append(f"{tag}{content}")
+    return "".join(merged_parts)
+
+
+def _normalize_prompt_text(prompt_text: str, speaker_id: int) -> str:
+    text = (prompt_text or "").strip()
+    if not text:
+        raise ValueError(f"S{speaker_id} prompt text is empty.")
+
+    expected_tag = f"[S{speaker_id}]"
+    if not text.lstrip().startswith(expected_tag):
+        text = f"{expected_tag} {text}"
+    return text
+
+
+def _build_prefixed_text(
+    dialogue_text: str,
+    prompt_text_map: dict[int, str],
+    cloned_speakers: list[int],
+) -> str:
+    prompt_prefix = "".join([prompt_text_map[speaker_id] for speaker_id in cloned_speakers])
+    return _merge_consecutive_speaker_tags(prompt_prefix + dialogue_text)
+
+
+def _encode_reference_audio_codes(
+    processor,
+    clone_wavs: list[torch.Tensor],
+    cloned_speakers: list[int],
+    speaker_count: int,
+    sample_rate: int,
+) -> list[Optional[torch.Tensor]]:
+    encoded_list = processor.encode_audios_from_wav(clone_wavs, sampling_rate=sample_rate)
+    reference_audio_codes: list[Optional[torch.Tensor]] = [None for _ in range(speaker_count)]
+    for speaker_id, audio_codes in zip(cloned_speakers, encoded_list):
+        reference_audio_codes[speaker_id - 1] = audio_codes
+    return reference_audio_codes
+
+
+def build_conversation(
+    dialogue_text: str,
+    reference_audio_codes: list[Optional[torch.Tensor]],
+    prompt_audio: torch.Tensor | None,
+    processor,
+):
+    if prompt_audio is None:
+        return [[processor.build_user_message(text=dialogue_text)]], "generation", "Generation"
+
+    user_message = processor.build_user_message(
+        text=dialogue_text,
+        reference=reference_audio_codes,
+    )
+    return (
+        [
+            [
+                user_message,
+                processor.build_assistant_message(audio_codes_list=[prompt_audio]),
+            ],
+        ],
+        "continuation",
+        "voice_clone_and_continuation",
+    )
+
+
+def run_inference(speaker_count: int, *all_inputs):
+    speaker_count = int(speaker_count)
+    speaker_count = max(MIN_SPEAKERS, min(MAX_SPEAKERS, speaker_count))
+
+    reference_audio_values = all_inputs[:MAX_SPEAKERS]
+    prompt_text_values = all_inputs[MAX_SPEAKERS : 2 * MAX_SPEAKERS]
+    dialogue_text = all_inputs[2 * MAX_SPEAKERS]
+    text_normalize, sample_rate_normalize, temperature, top_p, top_k, repetition_penalty, max_new_tokens, model_path, codec_path, device, attn_implementation = all_inputs[
+        2 * MAX_SPEAKERS + 1 :
+    ]
+
+    started_at = time.monotonic()
+    model, processor, torch_device, sample_rate = load_backend(
+        model_path=str(model_path),
+        codec_path=str(codec_path),
+        device_str=str(device),
+        attn_implementation=str(attn_implementation),
+    )
+
+    text_normalize = bool(text_normalize)
+    sample_rate_normalize = bool(sample_rate_normalize)
+
+    normalized_dialogue = str(dialogue_text or "").strip()
+    if text_normalize:
+        normalized_dialogue = normalize_text(normalized_dialogue)
+    normalized_dialogue = _validate_dialogue_text(normalized_dialogue, speaker_count)
+
+    cloned_speakers: list[int] = []
+    loaded_clone_wavs: list[tuple[torch.Tensor, int]] = []
+    prompt_text_map: dict[int, str] = {}
+    for idx in range(speaker_count):
+        ref_audio = reference_audio_values[idx]
+        prompt_text = str(prompt_text_values[idx] or "").strip()
+
+        has_reference = bool(ref_audio)
+        has_prompt_text = bool(prompt_text)
+        if has_reference != has_prompt_text:
+            raise ValueError(
+                f"S{idx + 1} must provide both reference audio and prompt text together."
+            )
+
+        if has_reference:
+            speaker_id = idx + 1
+            ref_audio_path = str(ref_audio)
+            cloned_speakers.append(speaker_id)
+            loaded_clone_wavs.append(_load_audio(ref_audio_path))
+            prompt_text_map[speaker_id] = _normalize_prompt_text(prompt_text, speaker_id)
+
+    prompt_audio: Optional[torch.Tensor] = None
+    reference_audio_codes: list[Optional[torch.Tensor]] = []
+    conversation_text = normalized_dialogue
+    if cloned_speakers:
+        conversation_text = _build_prefixed_text(
+            dialogue_text=normalized_dialogue,
+            prompt_text_map=prompt_text_map,
+            cloned_speakers=cloned_speakers,
+        )
+        if text_normalize:
+            conversation_text = normalize_text(conversation_text)
+        conversation_text = _validate_dialogue_text(conversation_text, speaker_count)
+
+        if sample_rate_normalize:
+            min_sr = min(sr for _, sr in loaded_clone_wavs)
+        else:
+            min_sr = None
+
+        clone_wavs: list[torch.Tensor] = []
+        for wav, orig_sr in loaded_clone_wavs:
+            processed_wav = wav
+            current_sr = int(orig_sr)
+            if min_sr is not None:
+                processed_wav = _resample_wav(processed_wav, current_sr, int(min_sr))
+                current_sr = int(min_sr)
+            processed_wav = _resample_wav(processed_wav, current_sr, sample_rate)
+            clone_wavs.append(processed_wav)
+
+        reference_audio_codes = _encode_reference_audio_codes(
+            processor=processor,
+            clone_wavs=clone_wavs,
+            cloned_speakers=cloned_speakers,
+            speaker_count=speaker_count,
+            sample_rate=sample_rate,
+        )
+        concat_prompt_wav = torch.cat(clone_wavs, dim=-1)
+        prompt_audio = processor.encode_audios_from_wav([concat_prompt_wav], sampling_rate=sample_rate)[0]
+
+    conversations, mode, mode_name = build_conversation(
+        dialogue_text=conversation_text,
+        reference_audio_codes=reference_audio_codes,
+        prompt_audio=prompt_audio,
+        processor=processor,
+    )
+
+    batch = processor(conversations, mode=mode)
+    input_ids = batch["input_ids"].to(torch_device)
+    attention_mask = batch["attention_mask"].to(torch_device)
+
+    with torch.no_grad():
+        outputs = model.generate(
+            input_ids=input_ids,
+            attention_mask=attention_mask,
+            max_new_tokens=int(max_new_tokens),
+            audio_temperature=float(temperature),
+            audio_top_p=float(top_p),
+            audio_top_k=int(top_k),
+            audio_repetition_penalty=float(repetition_penalty),
+        )
+
+    messages = processor.decode(outputs)
+    if not messages or messages[0] is None:
+        raise RuntimeError("The model did not return a decodable audio result.")
+
+    audio = messages[0].audio_codes_list[0]
+    if isinstance(audio, torch.Tensor):
+        audio_np = audio.detach().float().cpu().numpy()
+    else:
+        audio_np = np.asarray(audio, dtype=np.float32)
+
+    if audio_np.ndim > 1:
+        audio_np = audio_np.reshape(-1)
+    audio_np = audio_np.astype(np.float32, copy=False)
+
+    clone_summary = "none" if not cloned_speakers else ",".join([f"S{i}" for i in cloned_speakers])
+    elapsed = time.monotonic() - started_at
+    status = (
+        f"Done | mode={mode_name} | speakers={speaker_count} | cloned={clone_summary} | elapsed={elapsed:.2f}s | "
+        f"text_normalize={text_normalize}, sample_rate_normalize={sample_rate_normalize} | "
+        f"max_new_tokens={int(max_new_tokens)}, "
+        f"audio_temperature={float(temperature):.2f}, audio_top_p={float(top_p):.2f}, "
+        f"audio_top_k={int(top_k)}, audio_repetition_penalty={float(repetition_penalty):.2f}"
+    )
+    return (sample_rate, audio_np), status
+
+
+def build_demo(args: argparse.Namespace):
+    custom_css = """
+    :root {
+      --bg: #f6f7f8;
+      --panel: #ffffff;
+      --ink: #111418;
+      --muted: #4d5562;
+      --line: #e5e7eb;
+      --accent: #0f766e;
+    }
+    .gradio-container {
+      background: linear-gradient(180deg, #f7f8fa 0%, #f3f5f7 100%);
+      color: var(--ink);
+    }
+    .app-card {
+      border: 1px solid var(--line);
+      border-radius: 16px;
+      background: var(--panel);
+      padding: 14px;
+    }
+    .app-title {
+      font-size: 22px;
+      font-weight: 700;
+      margin-bottom: 6px;
+      letter-spacing: 0.2px;
+    }
+    .app-subtitle {
+      color: var(--muted);
+      font-size: 14px;
+      margin-bottom: 8px;
+    }
+    #output_panel {
+      overflow: hidden !important;
+    }
+    #output_audio {
+      padding-bottom: 24px;
+      margin-bottom: 0;
+      overflow: hidden !important;
+    }
+    #output_audio > .wrap,
+    #output_audio .wrap,
+    #output_audio .audio-container,
+    #output_audio .block {
+      overflow: hidden !important;
+    }
+    #output_audio .audio-container {
+      padding-bottom: 10px;
+      min-height: 96px;
+    }
+    #output_audio_spacer {
+      height: 12px;
+    }
+    #output_status {
+      margin-top: 0;
+    }
+    #run-btn {
+      background: var(--accent);
+      border: none;
+    }
+    """
+
+    with gr.Blocks(title="MOSS-TTSD Demo", css=custom_css) as demo:
+        gr.Markdown(
+            """
+            <div class="app-card">
+              <div class="app-title">MOSS-TTSD</div>
+              <div class="app-subtitle">Multi-speaker dialogue synthesis with optional per-speaker voice cloning.</div>
+            </div>
+            """
+        )
+
+        speaker_panels: list[gr.Group] = []
+        speaker_refs = []
+        speaker_prompts = []
+
+        with gr.Row(equal_height=False):
+            with gr.Column(scale=3):
+                speaker_count = gr.Slider(
+                    minimum=MIN_SPEAKERS,
+                    maximum=MAX_SPEAKERS,
+                    step=1,
+                    value=2,
+                    label="Speaker Count",
+                    info="Default 2 speakers. Minimum 1, maximum 5.",
+                )
+
+                gr.Markdown("### Voice Cloning (Optional, placed first)")
+                gr.Markdown(
+                    "If you provide reference audio for a speaker, you must also provide that speaker's prompt text. "
+                    "Prompt text may omit [Sx]; the app will auto-prepend it."
+                )
+
+                for idx in range(1, MAX_SPEAKERS + 1):
+                    with gr.Group(visible=idx <= 2) as panel:
+                        speaker_ref = gr.Audio(
+                            label=f"S{idx} Reference Audio (Optional)",
+                            type="filepath",
+                        )
+                        speaker_prompt = gr.Textbox(
+                            label=f"S{idx} Prompt Text (Required with reference audio)",
+                            lines=2,
+                            placeholder=f"Example: [S{idx}] This is a prompt line for S{idx}.",
+                        )
+                    speaker_panels.append(panel)
+                    speaker_refs.append(speaker_ref)
+                    speaker_prompts.append(speaker_prompt)
+
+                gr.Markdown("### Multi-turn Dialogue")
+                dialogue_text = gr.Textbox(
+                    label="Dialogue Text",
+                    lines=12,
+                    placeholder=(
+                        "Use explicit tags in a single box, e.g.\n"
+                        "[S1] Hello.\n"
+                        "[S2] Hi, how are you?\n"
+                        "[S1] Great, let's continue."
+                    ),
+                )
+                gr.Markdown(
+                    "Without any reference audio, the model runs in generation mode. "
+                    "Once any reference audio is provided, the model switches to voice-clone continuation mode."
+                )
+
+                with gr.Accordion("Sampling Parameters (Audio)", open=True):
+                    gr.Markdown(
+                        "- `text_normalize`: Normalize input text (**recommended to always enable**).\n"
+                        "- `sample_rate_normalize`: Resample prompt audios to the lowest sample rate before encoding "
+                        "(**recommended when using 2 or more speakers**)."
+                    )
+                    text_normalize = gr.Checkbox(
+                        value=True,
+                        label="text_normalize",
+                    )
+                    sample_rate_normalize = gr.Checkbox(
+                        value=False,
+                        label="sample_rate_normalize",
+                    )
+                    temperature = gr.Slider(
+                        minimum=0.1,
+                        maximum=3.0,
+                        step=0.05,
+                        value=1.1,
+                        label="temperature",
+                    )
+                    top_p = gr.Slider(
+                        minimum=0.1,
+                        maximum=1.0,
+                        step=0.01,
+                        value=0.9,
+                        label="top_p",
+                    )
+                    top_k = gr.Slider(
+                        minimum=1,
+                        maximum=200,
+                        step=1,
+                        value=50,
+                        label="top_k",
+                    )
+                    repetition_penalty = gr.Slider(
+                        minimum=0.8,
+                        maximum=2.0,
+                        step=0.05,
+                        value=1.1,
+                        label="repetition_penalty",
+                    )
+                    max_new_tokens = gr.Slider(
+                        minimum=256,
+                        maximum=8192,
+                        step=128,
+                        value=DEFAULT_MAX_NEW_TOKENS,
+                        label="max_new_tokens",
+                    )
+
+                run_btn = gr.Button("Generate Dialogue Audio", variant="primary", elem_id="run-btn")
+
+            with gr.Column(scale=2, elem_id="output_panel"):
+                output_audio = gr.Audio(label="Output Audio", type="numpy", elem_id="output_audio")
+                gr.HTML("", elem_id="output_audio_spacer")
+                status = gr.Textbox(label="Status", lines=4, interactive=False, elem_id="output_status")
+                preset_examples = gr.Dataframe(
+                    headers=["Field", "Value (click any row to fill inputs)"],
+                    value=PRESET_TABLE_ROWS,
+                    datatype=["str", "str"],
+                    row_count=(len(PRESET_TABLE_ROWS), "fixed"),
+                    col_count=(2, "fixed"),
+                    interactive=False,
+                    wrap=True,
+                    label="Preset Examples",
+                )
+
+        speaker_count.change(
+            fn=update_speaker_panels,
+            inputs=[speaker_count],
+            outputs=speaker_panels,
+        )
+        preset_examples.select(
+            fn=apply_preset_selection,
+            outputs=[
+                speaker_count,
+                speaker_refs[0],
+                speaker_prompts[0],
+                speaker_refs[1],
+                speaker_prompts[1],
+                dialogue_text,
+                *speaker_panels,
+            ],
+        )
+
+        run_btn.click(
+            fn=lambda speaker_count, *inputs: run_inference(
+                speaker_count,
+                *inputs,
+                args.model_path,
+                args.codec_path,
+                args.device,
+                args.attn_implementation,
+            ),
+            inputs=[
+                speaker_count,
+                *speaker_refs,
+                *speaker_prompts,
+                dialogue_text,
+                text_normalize,
+                sample_rate_normalize,
+                temperature,
+                top_p,
+                top_k,
+                repetition_penalty,
+                max_new_tokens,
+            ],
+            outputs=[output_audio, status],
+        )
+    return demo
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description="MOSS-TTSD Gradio Demo")
+    parser.add_argument("--model_path", type=str, default=MODEL_PATH)
+    parser.add_argument("--codec_path", type=str, default=CODEC_MODEL_PATH)
+    parser.add_argument("--device", type=str, default="cuda:0")
+    parser.add_argument("--attn_implementation", type=str, default=DEFAULT_ATTN_IMPLEMENTATION)
+    parser.add_argument("--host", type=str, default="0.0.0.0")
+    parser.add_argument("--port", type=int, default=7863)
+    parser.add_argument("--share", action="store_true")
+    args = parser.parse_args()
+
+    runtime_device = torch.device(args.device if torch.cuda.is_available() else "cpu")
+    runtime_dtype = torch.bfloat16 if runtime_device.type == "cuda" else torch.float32
+    args.attn_implementation = resolve_attn_implementation(
+        requested=args.attn_implementation,
+        device=runtime_device,
+        dtype=runtime_dtype,
+    ) or "none"
+    print(f"[INFO] Using attn_implementation={args.attn_implementation}", flush=True)
+
+    preload_started_at = time.monotonic()
+    print(
+        f"[Startup] Preloading backend: model={args.model_path}, codec={args.codec_path}, "
+        f"device={args.device}, attn={args.attn_implementation}",
+        flush=True,
+    )
+    load_backend(
+        model_path=args.model_path,
+        codec_path=args.codec_path,
+        device_str=args.device,
+        attn_implementation=args.attn_implementation,
+    )
+    print(
+        f"[Startup] Backend preload finished in {time.monotonic() - preload_started_at:.2f}s",
+        flush=True,
+    )
+
+    demo = build_demo(args)
+    demo.queue(default_concurrency_limit=2).launch(
+        server_name=args.host,
+        server_port=args.port,
+        share=args.share,
+    )
+
+
+if __name__ == "__main__":
+    main()

clis/moss_voice_generator_app.py

@@ -0,0 +1,410 @@
+import argparse
+import functools
+import importlib.util
+import json
+from pathlib import Path
+import re
+import time
+
+import gradio as gr
+import numpy as np
+import torch
+from transformers import AutoModel, AutoProcessor
+
+# Disable the broken cuDNN SDPA backend
+torch.backends.cuda.enable_cudnn_sdp(False)
+# Keep these enabled as fallbacks
+torch.backends.cuda.enable_flash_sdp(True)
+torch.backends.cuda.enable_mem_efficient_sdp(True)
+torch.backends.cuda.enable_math_sdp(True)
+
+MODEL_PATH = "OpenMOSS-Team/MOSS-VoiceGenerator"
+DEFAULT_ATTN_IMPLEMENTATION = "auto"
+DEFAULT_MAX_NEW_TOKENS = 4096
+EXAMPLE_TEXTS_JSONL_PATH = (
+    Path(__file__).resolve().parent.parent / "assets" / "text" / "moss_voice_generator_example_texts.jsonl"
+)
+
+
+def _parse_example_id(example_id: str) -> tuple[str, int] | None:
+    matched = re.fullmatch(r"(zh|en)/(\d+)", (example_id or "").strip())
+    if matched is None:
+        return None
+    return matched.group(1), int(matched.group(2))
+
+
+def build_example_rows() -> list[tuple[str, str, str]]:
+    rows: list[tuple[str, int, str, str]] = []
+    with open(EXAMPLE_TEXTS_JSONL_PATH, "r", encoding="utf-8") as f:
+        for line in f:
+            if not line.strip():
+                continue
+            sample = json.loads(line)
+            parsed = _parse_example_id(sample.get("id", ""))
+            if parsed is None:
+                continue
+
+            language, index = parsed
+            instruction = str(sample.get("instruction", "")).strip()
+            text = str(sample.get("text", "")).strip()
+            rows.append((language, index, instruction, text))
+
+    language_order = {"zh": 0, "en": 1}
+    rows.sort(key=lambda item: (language_order.get(item[0], 99), item[1]))
+    return [(f"{language}/{index}", instruction, text) for language, index, instruction, text in rows]
+
+
+EXAMPLE_ROWS = build_example_rows()
+
+
+def apply_example_selection(evt: gr.SelectData):
+    if evt is None or evt.index is None:
+        return gr.update(), gr.update()
+
+    if isinstance(evt.index, (tuple, list)):
+        row_idx = int(evt.index[0])
+    else:
+        row_idx = int(evt.index)
+
+    if row_idx < 0 or row_idx >= len(EXAMPLE_ROWS):
+        return gr.update(), gr.update()
+
+    _, instruction_value, text_value = EXAMPLE_ROWS[row_idx]
+    return instruction_value, text_value
+
+
+def resolve_attn_implementation(requested: str, device: torch.device, dtype: torch.dtype) -> str | None:
+    requested_norm = (requested or "").strip().lower()
+
+    if requested_norm in {"none"}:
+        return None
+
+    if requested_norm not in {"", "auto"}:
+        return requested
+
+    # Prefer FlashAttention 2 when package + device conditions are met.
+    if (
+        device.type == "cuda"
+        and importlib.util.find_spec("flash_attn") is not None
+        and dtype in {torch.float16, torch.bfloat16}
+    ):
+        major, _ = torch.cuda.get_device_capability(device)
+        if major >= 8:
+            return "flash_attention_2"
+
+    # CUDA fallback: use PyTorch SDPA kernels.
+    if device.type == "cuda":
+        return "sdpa"
+
+    # CPU fallback.
+    return "eager"
+
+
+@functools.lru_cache(maxsize=1)
+def load_backend(model_path: str, device_str: str, attn_implementation: str):
+    device = torch.device(device_str if torch.cuda.is_available() else "cpu")
+    dtype = torch.bfloat16 if device.type == "cuda" else torch.float32
+    resolved_attn_implementation = resolve_attn_implementation(
+        requested=attn_implementation,
+        device=device,
+        dtype=dtype,
+    )
+
+    processor = AutoProcessor.from_pretrained(
+        model_path,
+        trust_remote_code=True,
+        normalize_inputs=True,
+    )
+    if hasattr(processor, "audio_tokenizer"):
+        processor.audio_tokenizer = processor.audio_tokenizer.to(device)
+
+    model_kwargs = {
+        "trust_remote_code": True,
+        "torch_dtype": dtype,
+    }
+    if resolved_attn_implementation:
+        model_kwargs["attn_implementation"] = resolved_attn_implementation
+
+    model = AutoModel.from_pretrained(model_path, **model_kwargs).to(device)
+    model.eval()
+
+    sample_rate = int(getattr(processor.model_config, "sampling_rate", 24000))
+    return model, processor, device, sample_rate
+
+
+def build_conversation(text: str, instruction: str, processor):
+    text = (text or "").strip()
+    instruction = (instruction or "").strip()
+    if not text:
+        raise ValueError("Please enter text to synthesize.")
+    if not instruction:
+        raise ValueError("Please enter a voice instruction.")
+
+    return [[processor.build_user_message(text=text, instruction=instruction)]]
+
+
+def run_inference(
+    text: str,
+    instruction: str,
+    temperature: float,
+    top_p: float,
+    top_k: int,
+    repetition_penalty: float,
+    max_new_tokens: int,
+    model_path: str,
+    device: str,
+    attn_implementation: str,
+):
+    started_at = time.monotonic()
+    model, processor, torch_device, sample_rate = load_backend(
+        model_path=model_path,
+        device_str=device,
+        attn_implementation=attn_implementation,
+    )
+
+    conversations = build_conversation(
+        text=text,
+        instruction=instruction,
+        processor=processor,
+    )
+
+    batch = processor(conversations, mode="generation")
+    input_ids = batch["input_ids"].to(torch_device)
+    attention_mask = batch["attention_mask"].to(torch_device)
+
+    with torch.no_grad():
+        outputs = model.generate(
+            input_ids=input_ids,
+            attention_mask=attention_mask,
+            max_new_tokens=int(max_new_tokens),
+            audio_temperature=float(temperature),
+            audio_top_p=float(top_p),
+            audio_top_k=int(top_k),
+            audio_repetition_penalty=float(repetition_penalty),
+        )
+
+    messages = processor.decode(outputs)
+    if not messages or messages[0] is None:
+        raise RuntimeError("The model did not return a decodable audio result.")
+
+    audio = messages[0].audio_codes_list[0]
+    if isinstance(audio, torch.Tensor):
+        audio_np = audio.detach().float().cpu().numpy()
+    else:
+        audio_np = np.asarray(audio, dtype=np.float32)
+
+    if audio_np.ndim > 1:
+        audio_np = audio_np.reshape(-1)
+    audio_np = audio_np.astype(np.float32, copy=False)
+
+    elapsed = time.monotonic() - started_at
+    status = (
+        f"Done | elapsed: {elapsed:.2f}s | "
+        f"max_new_tokens={int(max_new_tokens)}, "
+        f"audio_temperature={float(temperature):.2f}, audio_top_p={float(top_p):.2f}, "
+        f"audio_top_k={int(top_k)}, audio_repetition_penalty={float(repetition_penalty):.2f}"
+    )
+    return (sample_rate, audio_np), status
+
+
+def build_demo(args: argparse.Namespace):
+    custom_css = """
+    :root {
+      --bg: #f6f7f8;
+      --panel: #ffffff;
+      --ink: #111418;
+      --muted: #4d5562;
+      --line: #e5e7eb;
+      --accent: #0f766e;
+    }
+    .gradio-container {
+      background: linear-gradient(180deg, #f7f8fa 0%, #f3f5f7 100%);
+      color: var(--ink);
+    }
+    .app-card {
+      border: 1px solid var(--line);
+      border-radius: 16px;
+      background: var(--panel);
+      padding: 14px;
+    }
+    .app-title {
+      font-size: 22px;
+      font-weight: 700;
+      margin-bottom: 6px;
+      letter-spacing: 0.2px;
+    }
+    .app-subtitle {
+      color: var(--muted);
+      font-size: 14px;
+      margin-bottom: 8px;
+    }
+    #output_audio {
+      padding-bottom: 12px;
+      margin-bottom: 8px;
+      overflow: hidden !important;
+    }
+    #output_audio > .wrap {
+      overflow: hidden !important;
+    }
+    #output_audio audio {
+      margin-bottom: 6px;
+    }
+    #run-btn {
+      background: var(--accent);
+      border: none;
+    }
+    """
+
+    with gr.Blocks(title="MOSS-VoiceGenerator Demo", css=custom_css) as demo:
+        gr.Markdown(
+            """
+            <div class="app-card">
+              <div class="app-title">MOSS-VoiceGenerator</div>
+              <div class="app-subtitle">Design expressive voices from instruction + text without reference audio.</div>
+            </div>
+            """
+        )
+
+        with gr.Row(equal_height=False):
+            with gr.Column(scale=3):
+                instruction = gr.Textbox(
+                    label="Voice Instruction",
+                    lines=5,
+                    placeholder="Example: Warm, gentle female narrator voice with calm pacing and clear articulation.",
+                )
+                text = gr.Textbox(
+                    label="Text",
+                    lines=8,
+                    placeholder="Enter the text content to synthesize with the instruction-defined voice.",
+                )
+
+                with gr.Accordion("Sampling Parameters (Audio)", open=True):
+                    temperature = gr.Slider(
+                        minimum=0.1,
+                        maximum=3.0,
+                        step=0.05,
+                        value=1.5,
+                        label="temperature",
+                    )
+                    top_p = gr.Slider(
+                        minimum=0.1,
+                        maximum=1.0,
+                        step=0.01,
+                        value=0.6,
+                        label="top_p",
+                    )
+                    top_k = gr.Slider(
+                        minimum=1,
+                        maximum=200,
+                        step=1,
+                        value=50,
+                        label="top_k",
+                    )
+                    repetition_penalty = gr.Slider(
+                        minimum=0.8,
+                        maximum=2.0,
+                        step=0.05,
+                        value=1.1,
+                        label="repetition_penalty",
+                    )
+                    max_new_tokens = gr.Slider(
+                        minimum=256,
+                        maximum=8192,
+                        step=128,
+                        value=DEFAULT_MAX_NEW_TOKENS,
+                        label="max_new_tokens",
+                    )
+
+                run_btn = gr.Button("Generate Voice", variant="primary", elem_id="run-btn")
+
+            with gr.Column(scale=2):
+                output_audio = gr.Audio(label="Output Audio", type="numpy", elem_id="output_audio")
+                status = gr.Textbox(label="Status", lines=4, interactive=False)
+                examples_table = gr.Dataframe(
+                    headers=["Voice Instruction", "Example Text"],
+                    value=[[example_instruction, example_text] for _, example_instruction, example_text in EXAMPLE_ROWS],
+                    datatype=["str", "str"],
+                    row_count=(len(EXAMPLE_ROWS), "fixed"),
+                    col_count=(2, "fixed"),
+                    interactive=False,
+                    wrap=True,
+                    label="Examples (click a row to fill inputs)",
+                )
+
+        examples_table.select(
+            fn=apply_example_selection,
+            inputs=[],
+            outputs=[instruction, text],
+        )
+
+        run_btn.click(
+            fn=lambda text, instruction, temperature, top_p, top_k, repetition_penalty, max_new_tokens: run_inference(
+                text=text,
+                instruction=instruction,
+                temperature=temperature,
+                top_p=top_p,
+                top_k=top_k,
+                repetition_penalty=repetition_penalty,
+                max_new_tokens=max_new_tokens,
+                model_path=args.model_path,
+                device=args.device,
+                attn_implementation=args.attn_implementation,
+            ),
+            inputs=[
+                text,
+                instruction,
+                temperature,
+                top_p,
+                top_k,
+                repetition_penalty,
+                max_new_tokens,
+            ],
+            outputs=[output_audio, status],
+        )
+    return demo
+
+
+def main():
+    parser = argparse.ArgumentParser(description="MOSS-VoiceGenerator Gradio Demo")
+    parser.add_argument("--model_path", type=str, default=MODEL_PATH)
+    parser.add_argument("--device", type=str, default="cuda:0")
+    parser.add_argument("--attn_implementation", type=str, default=DEFAULT_ATTN_IMPLEMENTATION)
+    parser.add_argument("--host", type=str, default="0.0.0.0")
+    parser.add_argument("--port", type=int, default=7862)
+    parser.add_argument("--share", action="store_true")
+    args = parser.parse_args()
+
+    runtime_device = torch.device(args.device if torch.cuda.is_available() else "cpu")
+    runtime_dtype = torch.bfloat16 if runtime_device.type == "cuda" else torch.float32
+    args.attn_implementation = resolve_attn_implementation(
+        requested=args.attn_implementation,
+        device=runtime_device,
+        dtype=runtime_dtype,
+    ) or "none"
+    print(f"[INFO] Using attn_implementation={args.attn_implementation}", flush=True)
+
+    preload_started_at = time.monotonic()
+    print(
+        f"[Startup] Preloading backend: model={args.model_path}, device={args.device}, attn={args.attn_implementation}",
+        flush=True,
+    )
+    load_backend(
+        model_path=args.model_path,
+        device_str=args.device,
+        attn_implementation=args.attn_implementation,
+    )
+    print(
+        f"[Startup] Backend preload finished in {time.monotonic() - preload_started_at:.2f}s",
+        flush=True,
+    )
+
+    demo = build_demo(args)
+    demo.queue(max_size=16, default_concurrency_limit=1).launch(
+        server_name=args.host,
+        server_port=args.port,
+        share=args.share,
+    )
+
+
+if __name__ == "__main__":
+    main()

configs/llama_cpp/cpu-only.yaml

@@ -0,0 +1,45 @@
+# MOSS-TTS-Delay — llama.cpp Backend (CPU-only)
+#
+# For machines without GPU.  All computation runs on CPU.
+# Performance will be slower but no CUDA/GPU drivers required.
+#
+# Installation:
+#   pip install -e ".[llama-cpp]"
+#   pip install onnxruntime    # CPU-only ONNX Runtime
+
+# ── Model paths ──────────────────────────────────────────────────────────────
+
+backbone_gguf: weights/MOSS-TTS-GGUF/MOSS_TTS_Q4_K_M.gguf
+embedding_dir: weights/MOSS-TTS-GGUF/embeddings
+lm_head_dir: weights/MOSS-TTS-GGUF/lm_heads
+tokenizer_dir: weights/MOSS-TTS-GGUF/tokenizer
+
+# ── Audio tokenizer ──────────────────────────────────────────────────────────
+
+audio_backend: onnx
+audio_encoder_onnx: weights/MOSS-Audio-Tokenizer-ONNX/encoder.onnx
+audio_decoder_onnx: weights/MOSS-Audio-Tokenizer-ONNX/decoder.onnx
+
+# ── LM heads backend ────────────────────────────────────────────────────────
+
+heads_backend: numpy
+
+# ── Runtime settings ─────────────────────────────────────────────────────────
+
+n_ctx: 4096
+n_batch: 256
+n_threads: 8
+n_gpu_layers: 0
+max_new_tokens: 3072
+use_gpu_audio: false
+
+# ── Sampling parameters ──────────────────────────────────────────────────────
+
+text_temperature: 1.5
+text_top_p: 1.0
+text_top_k: 50
+
+audio_temperature: 1.7
+audio_top_p: 0.8
+audio_top_k: 25
+audio_repetition_penalty: 1.0

configs/llama_cpp/default.yaml

@@ -0,0 +1,70 @@
+# MOSS-TTS-Delay — llama.cpp Backend (ONNX audio, default)
+#
+# Torch-free minimal installation:
+#   pip install -e ".[llama-cpp-onnx]"
+#
+# Download pre-quantized weights:
+#   huggingface-cli download OpenMOSS-Team/MOSS-TTS-GGUF --local-dir weights/MOSS-TTS-GGUF
+#
+# Download ONNX audio tokenizer:
+#   huggingface-cli download OpenMOSS-Team/MOSS-Audio-Tokenizer-ONNX --local-dir weights/MOSS-Audio-Tokenizer-ONNX
+#
+# All paths are relative to the project root.
+
+# ── Model paths ──────────────────────────────────────────────────────────────
+
+# Pre-quantized GGUF backbone (Q4_K_M)
+backbone_gguf: weights/MOSS-TTS-GGUF/MOSS_TTS_Q4_K_M.gguf
+
+# Pre-extracted .npy embedding tables and LM head weights
+embedding_dir: weights/MOSS-TTS-GGUF/embeddings
+lm_head_dir: weights/MOSS-TTS-GGUF/lm_heads
+
+# Tokenizer directory (ships with GGUF repo)
+tokenizer_dir: weights/MOSS-TTS-GGUF/tokenizer
+
+# ── Audio tokenizer ──────────────────────────────────────────────────────────
+# We provide ONNX models only.  For TensorRT, build engines yourself
+# (see moss_audio_tokenizer/trt/build_engine.sh).
+
+audio_backend: onnx
+audio_encoder_onnx: weights/MOSS-Audio-Tokenizer-ONNX/encoder.onnx
+audio_decoder_onnx: weights/MOSS-Audio-Tokenizer-ONNX/decoder.onnx
+
+# ── LM heads backend ────────────────────────────────────────────────────────
+# "auto" = use torch if available, else numpy
+# "numpy" = force numpy (torch-free)
+# "torch" = force torch (error if unavailable)
+
+heads_backend: auto
+
+# ── Runtime settings ─────────────────────────────────────────────────────────
+
+n_ctx: 4096
+n_batch: 512
+n_threads: 4
+n_gpu_layers: -1
+max_new_tokens: 3072
+use_gpu_audio: true
+
+# ── KV cache / attention ────────────────────────────────────────────────────
+# Quantized KV cache saves VRAM.  q8_0 is nearly lossless; q4_0 needs eval.
+# Options: f32, f16, bf16, q8_0, q5_0, q4_0
+# kv_cache_type_k: f16
+# kv_cache_type_v: f16
+
+# Flash attention reduces peak VRAM during prefill.  "auto" lets llama.cpp
+# decide; "enabled" forces it on (recommended when CUDA is available).
+# Options: auto, enabled, disabled
+# flash_attn: auto
+
+# ── Sampling parameters ──────────────────────────────────────────────────────
+
+text_temperature: 1.5
+text_top_p: 1.0
+text_top_k: 50
+
+audio_temperature: 1.7
+audio_top_p: 0.8
+audio_top_k: 25
+audio_repetition_penalty: 1.0

configs/llama_cpp/trt-8gb.yaml

@@ -0,0 +1,68 @@
+# MOSS-TTS-Delay — 8 GB GPU config (low-memory mode)
+#
+# Uses staged loading (low_memory: true) to load/unload components per stage.
+# Peak VRAM ≈ 5.6 GB (backbone stage), well within 8 GB.
+#
+# Run with:
+#   PYTHONPATH=. python -m moss_tts_delay.llama_cpp \
+#       --config configs/llama_cpp/trt-8gb.yaml --text "Hello!" --profile
+#
+#
+# Measured on H100 (transferable to any 8 GB+ GPU):
+#   Backbone (Q4_K_M, 36 layers, KV@4096, flash_attn) ≈ 5.6 GB
+#   TRT encoder alone                                  ≈ 5.3 GB
+#   TRT decoder alone                                  ≈ 5.3 GB
+#   Peak (staged)                                      ≈ 5.6 GB
+
+# ── Model paths ──────────────────────────────────────────────────────────────
+
+backbone_gguf: weights/MOSS-TTS-GGUF/MOSS_TTS_Q4_K_M.gguf
+embedding_dir: weights/MOSS-TTS-GGUF/embeddings
+lm_head_dir: weights/MOSS-TTS-GGUF/lm_heads
+tokenizer_dir: weights/MOSS-TTS-GGUF/tokenizer
+
+# ── Audio tokenizer ──────────────────────────────────────────────────────────
+# Switch to onnx + audio_encoder_onnx/audio_decoder_onnx if TRT not available.
+
+audio_backend: trt
+audio_encoder_trt: weights/MOSS-Audio-Tokenizer-TRT/encoder.engine
+audio_decoder_trt: weights/MOSS-Audio-Tokenizer-TRT/decoder.engine
+
+# ── LM heads backend ────────────────────────────────────────────────────────
+# numpy = 0 GPU, ~3 GB RAM.  Mandatory for 8 GB to keep backbone headroom.
+
+heads_backend: numpy
+
+# ── Runtime settings ─────────────────────────────────────────────────────────
+# n_ctx=4096 keeps KV cache at ~576 MB (fp16).  Enough for most single
+# utterances.  If you hit context-length errors, increase to 6144/8192
+# but check that backbone still fits (KV grows ~144 MB per 1024 tokens).
+
+n_ctx: 4096
+n_batch: 512
+n_threads: 4
+n_gpu_layers: -1
+max_new_tokens: 3072
+use_gpu_audio: true
+low_memory: true
+
+# ── KV cache / attention ────────────────────────────────────────────────────
+# Quantized KV cache saves ~0.45 GB (q8_0) or ~0.72 GB (q4_0) at n_ctx=4096.
+# Options: f32, f16, bf16, q8_0, q5_0, q4_0
+kv_cache_type_k: f16
+kv_cache_type_v: f16
+
+# Flash attention reduces peak VRAM during prefill (0.5–2 GB for long prompts).
+# Options: auto, enabled, disabled
+flash_attn: enabled
+
+# ── Sampling parameters ──────────────────────────────────────────────────────
+
+text_temperature: 1.5
+text_top_p: 1.0
+text_top_k: 50
+
+audio_temperature: 1.7
+audio_top_p: 0.8
+audio_top_k: 25
+audio_repetition_penalty: 1.0

configs/llama_cpp/trt.yaml

@@ -0,0 +1,54 @@
+# MOSS-TTS-Delay — llama.cpp Backend (TensorRT audio, maximum performance)
+#
+# ⚠️  TRT engines are NOT provided pre-built — you must build them yourself
+# from the ONNX models.  TRT engines are tied to your specific GPU architecture
+# and TensorRT version.
+#
+# Requirements:
+#   pip install -e ".[llama-cpp-trt,llama-cpp-torch]"
+#
+# Build engines:
+#   bash moss_audio_tokenizer/trt/build_engine.sh \
+#       weights/MOSS-Audio-Tokenizer-ONNX/encoder.onnx \
+#       weights/MOSS-Audio-Tokenizer-ONNX/decoder.onnx \
+#       weights/MOSS-Audio-Tokenizer-TRT
+#
+# ⚠️  IMPORTANT: maxShapes in build_engine.sh controls the maximum audio length
+# your engine can handle.  See the script's comments for details.
+
+# ── Model paths ──────────────────────────────────────────────────────────────
+
+backbone_gguf: weights/MOSS-TTS-GGUF/MOSS_TTS_Q4_K_M.gguf
+embedding_dir: weights/MOSS-TTS-GGUF/embeddings
+lm_head_dir: weights/MOSS-TTS-GGUF/lm_heads
+tokenizer_dir: weights/MOSS-TTS-GGUF/tokenizer
+
+# ── Audio tokenizer ──────────────────────────────────────────────────────────
+
+audio_backend: trt
+audio_encoder_trt: weights/MOSS-Audio-Tokenizer-TRT/encoder.engine
+audio_decoder_trt: weights/MOSS-Audio-Tokenizer-TRT/decoder.engine
+
+# ── LM heads backend ────────────────────────────────────────────────────────
+
+heads_backend: auto
+
+# ── Runtime settings ─────────────────────────────────────────────────────────
+
+n_ctx: 4096
+n_batch: 512
+n_threads: 32
+n_gpu_layers: -1
+max_new_tokens: 3072
+use_gpu_audio: true
+
+# ── Sampling parameters ──────────────────────────────────────────────────────
+
+text_temperature: 1.5
+text_top_p: 1.0
+text_top_k: 50
+
+audio_temperature: 1.7
+audio_top_p: 0.8
+audio_top_k: 25
+audio_repetition_penalty: 1.0

docs/moss_sound_effect_model_card.md

@@ -0,0 +1,142 @@
+# MOSS-SoundEffect Model Card
+
+**MOSS-SoundEffect** is the **environment sound & sound effect generation model** in the **MOSS‑TTS Family**. It generates ambient soundscapes and concrete sound effects directly from text descriptions, and is designed to complement speech content with immersive context in production workflows.
+
+
+## 1. Overview
+
+### 1.1 TTS Family Positioning
+
+MOSS-SoundEffect is designed as an audio generation backbone for creating high-fidelity environmental and action sounds from text, serving both scalable content pipelines and a strong research baseline for controllable audio generation.
+
+**Design goals**
+* **Coverage & richness**: broad sound taxonomy with layered ambience and realistic texture
+* **Composability**: easy integration into creative pipelines (games/film/tools) and synthetic data generation setups
+
+
+### 1.2 Key Capabilities
+MOSS‑SoundEffect focuses on **contextual audio completion** beyond speech, enabling creators and systems to enrich scenes with believable acoustic environments and action‑level cues.
+
+**What it can generate**
+- **Natural environments**: e.g., “fresh snow crunching under footsteps.”
+- **Urban environments**: e.g., “a sports car roaring past on the highway.”
+- **Animals & creatures**: e.g., “early morning park with birds chirping in a quiet atmosphere.”
+- **Human actions**: e.g., “clear footsteps echoing on concrete at a steady rhythm.”
+
+**Why it matters**
+- Completes **scene immersion** for narrative content, film/TV, documentaries, games, and podcasts.
+- Supports **voice agents** and interactive systems that need ambient context, not just speech.
+- Acts as the **sound‑design layer** of the MOSS‑TTS Family’s end‑to‑end workflow.
+
+
+
+### 1.3 Model Architecture
+**MOSS-SoundEffect** employs the **MossTTSDelay** architecture (see [moss_tts_delay/README.md](../moss_tts_delay/README.md)), reusing the same discrete token generation backbone for audio synthesis. A text prompt (optionally with simple control tags such as **duration**) is tokenized and fed into the Delay-pattern autoregressive model to predict **RVQ audio tokens** over time. The generated tokens are then decoded by the audio tokenizer/vocoder to produce high-fidelity sound effects, enabling consistent quality and controllable length across diverse SFX categories.
+
+
+
+### 1.4 Released Models
+**Recommended decoding hyperparameters**
+| Model | audio_temperature | audio_top_p | audio_top_k | audio_repetition_penalty |
+|---|---:|---:|---:|---:|
+| **MOSS-SoundEffect** | 1.5 | 0.6 | 50 | 1.2 |
+
+
+## 2. Quick Start
+
+
+
+```python
+from pathlib import Path
+import importlib.util
+import torch
+import torchaudio
+from transformers import AutoModel, AutoProcessor
+# Disable the broken cuDNN SDPA backend
+torch.backends.cuda.enable_cudnn_sdp(False)
+# Keep these enabled as fallbacks
+torch.backends.cuda.enable_flash_sdp(True)
+torch.backends.cuda.enable_mem_efficient_sdp(True)
+torch.backends.cuda.enable_math_sdp(True)
+
+
+pretrained_model_name_or_path = "OpenMOSS-Team/MOSS-SoundEffect"
+device = "cuda" if torch.cuda.is_available() else "cpu"
+dtype = torch.bfloat16 if device == "cuda" else torch.float32
+
+def resolve_attn_implementation() -> str:
+    # Prefer FlashAttention 2 when package + device conditions are met.
+    if (
+        device == "cuda"
+        and importlib.util.find_spec("flash_attn") is not None
+        and dtype in {torch.float16, torch.bfloat16}
+    ):
+        major, _ = torch.cuda.get_device_capability()
+        if major >= 8:
+            return "flash_attention_2"
+
+    # CUDA fallback: use PyTorch SDPA kernels.
+    if device == "cuda":
+        return "sdpa"
+
+    # CPU fallback.
+    return "eager"
+
+
+attn_implementation = resolve_attn_implementation()
+print(f"[INFO] Using attn_implementation={attn_implementation}")
+
+processor = AutoProcessor.from_pretrained(
+    pretrained_model_name_or_path,
+    trust_remote_code=True,
+)
+processor.audio_tokenizer = processor.audio_tokenizer.to(device)
+
+text_1 = "雷声隆隆，雨声淅沥。"
+text_2 = "清晰脚步声在水泥地面回响，节奏稳定。"
+
+conversations = [
+    [processor.build_user_message(ambient_sound=text_1)],
+    [processor.build_user_message(ambient_sound=text_2)]
+]
+
+model = AutoModel.from_pretrained(
+    pretrained_model_name_or_path,
+    trust_remote_code=True,
+    attn_implementation=attn_implementation,
+    torch_dtype=dtype,
+).to(device)
+model.eval()
+
+batch_size = 1
+
+save_dir = Path("inference_root")
+save_dir.mkdir(exist_ok=True, parents=True)
+sample_idx = 0
+with torch.no_grad():
+    for start in range(0, len(conversations), batch_size):
+        batch_conversations = conversations[start : start + batch_size]
+        batch = processor(batch_conversations, mode="generation")
+        input_ids = batch["input_ids"].to(device)
+        attention_mask = batch["attention_mask"].to(device)
+
+        outputs = model.generate(
+            input_ids=input_ids,
+            attention_mask=attention_mask,
+            max_new_tokens=4096,
+        )
+
+        for message in processor.decode(outputs):
+            audio = message.audio_codes_list[0]
+            out_path = save_dir / f"sample{sample_idx}.wav"
+            sample_idx += 1
+            torchaudio.save(out_path, audio.unsqueeze(0), processor.model_config.sampling_rate)
+```
+
+### Input Types
+
+**UserMessage**
+| Field | Type | Required | Description |
+|---|---|---:|---|
+| `ambient_sound` | `str` | Yes | Description of environment sound & sound effect |
+| `tokens` | `int` | No | Expected number of audio tokens. **1s ≈ 12.5 tokens**. |

docs/moss_tts_model_card.md

@@ -0,0 +1,427 @@
+# MOSS-TTS Model Card
+
+**MOSS-TTS** is a next-generation, production-grade TTS foundation model focused on **voice cloning**, **ultra-long stable speech generation**, **token-level duration control**, **multilingual & code-switched synthesis**, and **fine-grained Pinyin/phoneme-level pronunciation control**. It is built on a clean autoregressive discrete-token recipe that emphasizes high-quality audio tokenization, large-scale diverse pre-training data, and efficient discrete token modeling.
+
+
+
+## 1. Overview
+
+### 1.1 TTS Family Positioning
+MOSS-TTS is the **flagship base model** in our open-source **TTS Family**. It is designed as a production-ready synthesis backbone that can serve as the primary high-quality engine for scalable voice applications, and as a strong research baseline for controllable TTS and discrete audio token modeling.
+
+**Design goals**
+- **Production readiness**: robust voice cloning with stable, on-brand speaker identity at scale
+- **Controllability**: duration and pronunciation controls that integrate into real workflows
+- **Long-form stability**: consistent identity and delivery for extended narration
+- **Multilingual coverage**: multilingual and code-switched synthesis as first-class capabilities
+
+
+
+### 1.2 Key Capabilities
+
+MOSS-TTS delivers state-of-the-art quality while providing the fine-grained controllability and long-form stability required for production-grade voice applications, from zero-shot cloning and hour-long narration to token- and phoneme-level control across multilingual and code-switched speech.
+
+* **State-of-the-art evaluation performance** — top-tier objective and subjective results across standard TTS benchmarks and in-house human preference testing, validating both fidelity and naturalness.
+* **Zero-shot Voice Cloning (Voice Clone)** — clone a target speaker’s timbre (and part of speaking style) from short reference audio, without speaker-specific fine-tuning.
+* **Ultra-long Speech Generation (up to 1 hour)** — support continuous long-form speech generation for up to one hour in a single run, designed for extended narration and long-session content creation.
+* **Token-level Duration Control** — control pacing, rhythm, pauses, and speaking rate at token resolution for precise alignment and expressive delivery.
+* **Phoneme-level Pronunciation Control** — supports:
+
+  * pure **Pinyin** input
+  * pure **IPA** phoneme input
+  * mixed **Chinese / English / Pinyin / IPA** input in any combination
+* **Multilingual support** — high-quality multilingual synthesis with robust generalization across languages and accents.
+* **Code-switching** — natural mixed-language generation within a single utterance (e.g., Chinese–English), with smooth transitions, consistent speaker identity, and pronunciation-aware rendering on both sides of the switch.
+
+
+
+### 1.3 Model Architecture
+
+MOSS-TTS includes **two complementary architectures**, both trained and released to explore different performance/latency tradeoffs and to support downstream research.
+
+**Architecture A: Delay Pattern (MossTTSDelay)**
+- Single Transformer backbone with **(n_vq + 1) heads**.
+- Uses **delay scheduling** for multi-codebook audio tokens.
+- Strong long-context stability, efficient inference, and production-friendly behavior.
+
+**Architecture B: Global Latent + Local Transformer (MossTTSLocal)**
+- Backbone produces a **global latent** per time step.
+- A lightweight **Local Transformer** emits a token block per step.
+- **Streaming-friendly** with simpler alignment (no delay scheduling).
+
+**Why train both?**
+- **Exploration of architectural potential** and validation across multiple generation paradigms.
+- **Different tradeoffs**: Delay pattern tends to be faster and more stable for long-form synthesis; Local is smaller and excels on objective benchmarks.
+- **Open-source value**: two strong baselines for research, ablation, and downstream innovation.
+
+For full details, see:
+- **`moss_tts_delay/README.md`**
+- **`moss_tts_local/README.md`**
+
+
+
+### 1.4 Released Models
+
+| Model | Description |
+|---|---|
+| **MossTTSDelay-8B** | **Recommended for production**. Faster inference, stronger long-context stability, and robust voice cloning quality. Best for large-scale deployment and long-form narration. |
+| **MossTTSLocal-1.7B** | **Recommended for evaluation and research**. Smaller model size with SOTA objective metrics. Great for quick experiments, ablations, and academic studies. |
+
+**Recommended decoding hyperparameters (per model)**
+
+| Model | audio_temperature | audio_top_p | audio_top_k | audio_repetition_penalty |
+|---|---:|---:|---:|---:|
+| **MossTTSDelay-8B** | 1.7 | 0.8 | 25 | 1.0 |
+| **MossTTSLocal-1.7B** | 1.0 | 0.95 | 50 | 1.1 |
+
+
+
+
+## 2. Quick Start
+
+> Tip: For production usage, prioritize **MossTTSDelay-8B**. The examples below use this model; **MossTTSLocal-1.7B** supports the same API, and a practical walkthrough is available in [moss_tts_local/README.md](../moss_tts_local/README.md).
+
+MOSS-TTS provides a convenient `generate` interface for rapid usage. The examples below cover:
+1. Direct generation (Chinese / English / Pinyin / IPA)
+2. Voice cloning
+3. Duration control
+
+```python
+from pathlib import Path
+import importlib.util
+import torch
+import torchaudio
+from transformers import AutoModel, AutoProcessor
+# Disable the broken cuDNN SDPA backend
+torch.backends.cuda.enable_cudnn_sdp(False)
+# Keep these enabled as fallbacks
+torch.backends.cuda.enable_flash_sdp(True)
+torch.backends.cuda.enable_mem_efficient_sdp(True)
+torch.backends.cuda.enable_math_sdp(True)
+
+
+pretrained_model_name_or_path = "OpenMOSS-Team/MOSS-TTS"
+device = "cuda" if torch.cuda.is_available() else "cpu"
+dtype = torch.bfloat16 if device == "cuda" else torch.float32
+
+def resolve_attn_implementation() -> str:
+    # Prefer FlashAttention 2 when package + device conditions are met.
+    if (
+        device == "cuda"
+        and importlib.util.find_spec("flash_attn") is not None
+        and dtype in {torch.float16, torch.bfloat16}
+    ):
+        major, _ = torch.cuda.get_device_capability()
+        if major >= 8:
+            return "flash_attention_2"
+
+    # CUDA fallback: use PyTorch SDPA kernels.
+    if device == "cuda":
+        return "sdpa"
+
+    # CPU fallback.
+    return "eager"
+
+
+attn_implementation = resolve_attn_implementation()
+print(f"[INFO] Using attn_implementation={attn_implementation}")
+
+processor = AutoProcessor.from_pretrained(
+    pretrained_model_name_or_path,
+    trust_remote_code=True,
+)
+processor.audio_tokenizer = processor.audio_tokenizer.to(device)
+
+text_1 = "亲爱的你，\n你好呀。\n\n今天，我想用最认真、最温柔的声音，对你说一些重要的话。\n这些话，像一颗小小的星星，希望能在你的心里慢慢发光。\n\n首先，我想祝你——\n每天都能平平安安、快快乐乐。\n\n希望你早上醒来的时候，\n窗外有光，屋子里很安静，\n你的心是轻轻的，没有着急，也没有害怕。\n\n希望你吃饭的时候胃口很好，\n走路的时候脚步稳稳，\n晚上睡觉的时候，能做一个又一个甜甜的梦。\n\n我希望你能一直保持好奇心。\n对世界充满问题，\n对天空、星星、花草、书本和故事感兴趣。\n当你问“为什么”的时候，\n希望总有人愿意认真地听你说话。\n\n我也希望你学会温柔。\n温柔地对待朋友，\n温柔地对待小动物，\n也温柔地对待自己。\n\n如果有一天你犯了错，\n请不要太快责怪自己，\n因为每一个认真成长的人，\n都会在路上慢慢学会更好的方法。\n\n愿你拥有勇气。\n当你站在陌生的地方时，\n当你第一次举手发言时，\n当你遇到困难、感到害怕的时候，\n希望你能轻轻地告诉自己：\n“我可以试一试。”\n\n就算没有一次成功，也没有关系。\n失败不是坏事，\n它只是告诉你，你正在努力。\n\n我希望你学会分享快乐。\n把开心的事情告诉别人，\n把笑声送给身边的人，\n因为快乐被分享的时候，\n会变得更大、更亮。\n\n如果有一天你感到难过，\n我希望你知道——\n难过并不丢脸，\n哭泣也不是软弱。\n\n愿你能找到一个安全的地方，\n慢慢把心里的话说出来，\n然后再一次抬起头，看见希望。\n\n我还希望你能拥有梦想。\n这个梦想也许很大，\n也许很小，\n也许现在还说不清楚。\n\n没关系。\n梦想会和你一起长大，\n在时间里慢慢变得清楚。\n\n最后，我想送你一个最最重要的祝福：\n\n愿你被世界温柔对待，\n也愿你成为一个温柔的人。\n\n愿你的每一天，\n都值得被记住，\n都值得被珍惜。\n\n亲爱的你，\n请记住，\n你是独一无二的，\n你已经很棒了，\n而你的未来，\n一定会慢慢变得闪闪发光。\n\n祝你健康、勇敢、幸福，\n祝你永远带着笑容向前走。"
+text_2 = "We stand on the threshold of the AI era.\nArtificial intelligence is no longer just a concept in laboratories, but is entering every industry, every creative endeavor, and every decision. It has learned to see, hear, speak, and think, and is beginning to become an extension of human capabilities. AI is not about replacing humans, but about amplifying human creativity, making knowledge more equitable, more efficient, and allowing imagination to reach further. A new era, jointly shaped by humans and intelligent systems, has arrived."
+text_3 = "nin2 hao3，qing3 wen4 nin2 lai2 zi4 na3 zuo4 cheng2 shi4？"
+text_4 = "nin2 hao3，qing4 wen3 nin2 lai2 zi4 na4 zuo3 cheng4 shi3？"
+text_5 = "您好，请问您来自哪 zuo4 cheng2 shi4？"
+text_6 = "/həloʊ, meɪ aɪ æsk wɪtʃ sɪti juː ɑːr frʌm?/"
+
+# Use audio from ./assets/audio to avoid downloading from the cloud.
+ref_audio_1 = "https://speech-demo.oss-cn-shanghai.aliyuncs.com/moss_tts_demo/tts_readme_demo/reference_zh.wav"
+ref_audio_2 = "https://speech-demo.oss-cn-shanghai.aliyuncs.com/moss_tts_demo/tts_readme_demo/reference_en.m4a"
+
+conversations = [
+    # Direct TTS (no reference)
+    [processor.build_user_message(text=text_1)],
+    [processor.build_user_message(text=text_2)],
+    # Pinyin or IPA input
+    [processor.build_user_message(text=text_3)],
+    [processor.build_user_message(text=text_4)],
+    [processor.build_user_message(text=text_5)],
+    [processor.build_user_message(text=text_6)],
+    # Voice cloning (with reference)
+    [processor.build_user_message(text=text_1, reference=[ref_audio_1])],
+    [processor.build_user_message(text=text_2, reference=[ref_audio_2])],
+    # Duration control
+    [processor.build_user_message(text=text_2, tokens=325)],
+    [processor.build_user_message(text=text_2, tokens=600)],
+]
+
+model = AutoModel.from_pretrained(
+    pretrained_model_name_or_path,
+    trust_remote_code=True,
+    attn_implementation=attn_implementation,
+    torch_dtype=dtype,
+).to(device)
+model.eval()
+
+batch_size = 1
+
+save_dir = Path("inference_root")
+save_dir.mkdir(exist_ok=True, parents=True)
+sample_idx = 0
+with torch.no_grad():
+    for start in range(0, len(conversations), batch_size):
+        batch_conversations = conversations[start : start + batch_size]
+        batch = processor(batch_conversations, mode="generation")
+        input_ids = batch["input_ids"].to(device)
+        attention_mask = batch["attention_mask"].to(device)
+
+        outputs = model.generate(
+            input_ids=input_ids,
+            attention_mask=attention_mask,
+            max_new_tokens=4096,
+        )
+
+        for message in processor.decode(outputs):
+            audio = message.audio_codes_list[0]
+            out_path = save_dir / f"sample{sample_idx}.wav"
+            sample_idx += 1
+            torchaudio.save(out_path, audio.unsqueeze(0), processor.model_config.sampling_rate)
+
+```
+
+### Continuation + Voice Cloning (Prefix Audio + Text)
+
+MOSS-TTS supports continuation-based cloning: provide a prefix audio clip in the assistant message, and make sure the **prefix transcript** is included in the text. The model continues in the same speaker identity and style.
+
+```python
+from pathlib import Path
+import importlib.util
+import torch
+import torchaudio
+from transformers import AutoModel, AutoProcessor
+# Disable the broken cuDNN SDPA backend
+torch.backends.cuda.enable_cudnn_sdp(False)
+# Keep these enabled as fallbacks
+torch.backends.cuda.enable_flash_sdp(True)
+torch.backends.cuda.enable_mem_efficient_sdp(True)
+torch.backends.cuda.enable_math_sdp(True)
+
+
+pretrained_model_name_or_path = "OpenMOSS-Team/MOSS-TTS"
+device = "cuda" if torch.cuda.is_available() else "cpu"
+dtype = torch.bfloat16 if device == "cuda" else torch.float32
+
+def resolve_attn_implementation() -> str:
+    # Prefer FlashAttention 2 when package + device conditions are met.
+    if (
+        device == "cuda"
+        and importlib.util.find_spec("flash_attn") is not None
+        and dtype in {torch.float16, torch.bfloat16}
+    ):
+        major, _ = torch.cuda.get_device_capability()
+        if major >= 8:
+            return "flash_attention_2"
+
+    # CUDA fallback: use PyTorch SDPA kernels.
+    if device == "cuda":
+        return "sdpa"
+
+    # CPU fallback.
+    return "eager"
+
+
+attn_implementation = resolve_attn_implementation()
+print(f"[INFO] Using attn_implementation={attn_implementation}")
+
+processor = AutoProcessor.from_pretrained(
+    pretrained_model_name_or_path,
+    trust_remote_code=True
+)
+processor.audio_tokenizer = processor.audio_tokenizer.to(device)
+
+text_1 = "亲爱的你，\n你好呀。\n\n今天，我想用最认真、最温柔的声音，对你说一些重要的话。\n这些话，像一颗小小的星星，希望能在你的心里慢慢发光。\n\n首先，我想祝你——\n每天都能平平安安、快快乐乐。\n\n希望你早上醒来的时候，\n窗外有光，屋子里很安静，\n你的心是轻轻的，没有着急，也没有害怕。\n\n希望你吃饭的时候胃口很好，\n走路的时候脚步稳稳，\n晚上睡觉的时候，能做一个又一个甜甜的梦。\n\n我希望你能一直保持好奇心。\n对世界充满问题，\n对天空、星星、花草、书本和故事感兴趣。\n当你问“为什么”的时候，\n希望总有人愿意认真地听你说话。\n\n我也希望你学会温柔。\n温柔地对待朋友，\n温柔地对待小动物，\n也温柔地对待自己。\n\n如果有一天你犯了错，\n请不要太快责怪自己，\n因为每一个认真成长的人，\n都会在路上慢慢学会更好的方法。\n\n愿你拥有勇气。\n当你站在陌生的地方时，\n当你第一次举手发言时，\n当你遇到困难、感到害怕的时候，\n希望你能轻轻地告诉自己：\n“我可以试一试。”\n\n就算没有一次成功，也没有关系。\n失败不是坏事，\n它只是告诉你，你正在努力。\n\n我希望你学会分享快乐。\n把开心的事情告诉别人，\n把笑声送给身边的人，\n因为快乐被分享的时候，\n会变得更大、更亮。\n\n如果有一天你感到难过，\n我希望你知道——\n难过��不丢脸，\n哭泣也不是软弱。\n\n愿你能找到一个安全的地方，\n慢慢把心里的话说出来，\n然后再一次抬起头，看见希望。\n\n我还希望你能拥有梦想。\n这个梦想也许很大，\n也许很小，\n也许现在还说不清楚。\n\n没关系。\n梦想会和你一起长大，\n在时间里慢慢变得清楚。\n\n最后，我想送你一个最最重要的祝福：\n\n愿你被世界温柔对待，\n也愿你成为一个温柔的人。\n\n愿你的每一天，\n都值得被记住，\n都值得被珍惜。\n\n亲爱的你，\n请记住，\n你是独一无二的，\n你已经很棒了，\n而你的未来，\n一定会慢慢变得闪闪发光。\n\n祝你健康、勇敢、幸福，\n祝你永远带着笑容向前走。"
+text_2 = "We stand on the threshold of the AI era.\nArtificial intelligence is no longer just a concept in laboratories, but is entering every industry, every creative endeavor, and every decision. It has learned to see, hear, speak, and think, and is beginning to become an extension of human capabilities. AI is not about replacing humans, but about amplifying human creativity, making knowledge more equitable, more efficient, and allowing imagination to reach further. A new era, jointly shaped by humans and intelligent systems, has arrived."
+ref_text_1 = "太阳系八大行星之一。"
+ref_text_2 = "But I really can't complain about not having a normal college experience to you."
+# Use audio from ./assets/audio to avoid downloading from the cloud.
+ref_audio_1 = "https://speech-demo.oss-cn-shanghai.aliyuncs.com/moss_tts_demo/tts_readme_demo/reference_zh.wav"
+ref_audio_2 = "https://speech-demo.oss-cn-shanghai.aliyuncs.com/moss_tts_demo/tts_readme_demo/reference_en.m4a"
+
+conversations = [
+    # Continuatoin only
+    [
+        processor.build_user_message(text=ref_text_1 + text_1),
+        processor.build_assistant_message(audio_codes_list=[ref_audio_1])
+    ],
+    # Continuation with voice cloning
+    [
+        processor.build_user_message(text=ref_text_2 + text_2, reference=[ref_audio_2]),
+        processor.build_assistant_message(audio_codes_list=[ref_audio_2])
+    ],
+]
+
+model = AutoModel.from_pretrained(
+    pretrained_model_name_or_path,
+    trust_remote_code=True,
+    attn_implementation=attn_implementation,
+    torch_dtype=dtype,
+).to(device)
+model.eval()
+
+batch_size = 1
+
+save_dir = Path("inference_root")
+save_dir.mkdir(exist_ok=True, parents=True)
+sample_idx = 0
+with torch.no_grad():
+    for start in range(0, len(conversations), batch_size):
+        batch_conversations = conversations[start : start + batch_size]
+        batch = processor(batch_conversations, mode="continuation")
+        input_ids = batch["input_ids"].to(device)
+        attention_mask = batch["attention_mask"].to(device)
+
+        outputs = model.generate(
+            input_ids=input_ids,
+            attention_mask=attention_mask,
+            max_new_tokens=4096,
+        )
+
+        for message in processor.decode(outputs):
+            audio = message.audio_codes_list[0]
+            out_path = save_dir / f"sample{sample_idx}.wav"
+            sample_idx += 1
+            torchaudio.save(out_path, audio.unsqueeze(0), processor.model_config.sampling_rate)
+
+```
+
+
+
+### Input Types
+
+**UserMessage**
+
+| Field | Type | Required | Description |
+|---|---|---:|---|
+| `text` | `str` | Yes | Text to synthesize. Supports Chinese, English, German, French, Spanish, Japanese, Korean, etc. Can mix raw text with Pinyin or IPA for pronunciation control. |
+| `reference` | `List[str]` | No | Reference audio for voice cloning. For current MOSS-TTS, **one audio** is expected in the list. |
+| `tokens` | `int` | No | Expected number of audio tokens. **1s ≈ 12.5 tokens**. |
+
+**AssistantMessage**
+
+| Field | Type | Required | Description |
+|---|---|---:|---|
+| `audio_codes_list` | `List[str]` | Only for continuation | Prefix audio for continuation-based cloning. Use audio file paths or URLs. |
+
+
+
+### Generation Hyperparameters
+
+| Parameter | Type | Default | Description |
+|---|---|---:|---|
+| `max_new_tokens` | `int` | — | Controls total generated audio tokens. Use duration rule: **1s ≈ 12.5 tokens**. |
+| `audio_temperature` | `float` | 1.7 | Higher values increase variation; lower values stabilize prosody. |
+| `audio_top_p` | `float` | 0.8 | Nucleus sampling cutoff. Lower values are more conservative. |
+| `audio_top_k` | `int` | 25 | Top-K sampling. Lower values tighten sampling space. |
+| `audio_repetition_penalty` | `float` | 1.0 | >1.0 discourages repeating patterns. |
+
+> Note: MOSS-TTS is a pretrained base model and is **sensitive to decoding hyperparameters**. See **Released Models** for recommended defaults.
+
+
+
+### Pinyin Input
+
+Use tone-numbered Pinyin such as `ni3 hao3 wo3 men1`. You can convert Chinese text with [pypinyin](https://github.com/mozillazg/python-pinyin), then adjust tones for pronunciation control.
+
+```python
+import re
+from pypinyin import pinyin, Style
+
+CN_PUNCT = r"，。！？；：、（）“”‘���"
+
+
+def fix_punctuation_spacing(s: str) -> str:
+    s = re.sub(rf"\s+([{CN_PUNCT}])", r"\1", s)
+    s = re.sub(rf"([{CN_PUNCT}])\s+", r"\1", s)
+    return s
+
+
+def zh_to_pinyin_tone3(text: str, strict: bool = True) -> str:
+    result = pinyin(
+        text,
+        style=Style.TONE3,
+        heteronym=False,
+        strict=strict,
+        errors="default",
+    )
+
+    s = " ".join(item[0] for item in result)
+    return fix_punctuation_spacing(s)
+
+text = zh_to_pinyin_tone3("您好，请问您来自哪座城市？")
+print(text)
+
+# Expected: nin2 hao3，qing3 wen4 nin2 lai2 zi4 na3 zuo4 cheng2 shi4？
+# Try: nin2 hao3，qing4 wen3 nin2 lai2 zi4 na4 zuo3 cheng4 shi3？
+```
+
+
+
+### IPA Input
+
+Use `/.../` to wrap IPA sequences so they are distinct from normal text. You can use [DeepPhonemizer](https://github.com/spring-media/DeepPhonemizer) to convert English paragraphs or words into IPA sequences.
+
+```python
+from dp.phonemizer import Phonemizer
+
+# Download a phonemizer checkpoint from https://public-asai-dl-models.s3.eu-central-1.amazonaws.com/DeepPhonemizer/en_us_cmudict_ipa_forward.pt
+model_path = "<path-to-phonemizer-checkpoint>"
+phonemizer = Phonemizer.from_checkpoint(model_path)
+
+english_texts = "Hello, may I ask which city you are from?"
+phoneme_outputs = phonemizer(
+    english_texts,
+    lang="en_us",
+    batch_size=8
+)
+model_input_text = f"/{phoneme_outputs}/"
+print(model_input_text)
+
+# Expected: /həloʊ, meɪ aɪ æsk wɪtʃ sɪti juː ɑːr frʌm?/
+```
+
+
+
+## 3. Evaluation
+MOSS-TTS achieved state-of-the-art results on the open-source zero-shot TTS benchmark Seed-TTS-eval, not only surpassing all open-source models but also rivaling the most powerful closed-source models.
+
+| Model | Params | Open-source | EN WER (%) ↓ | EN SIM (%) ↑ | ZH CER (%) ↓ | ZH SIM (%) ↑ |
+|---|---:|:---:|---:|---:|---:|---:|
+| DiTAR | 0.6B | ❌ | 1.69 | 73.5 | 1.02 | 75.3 |
+| FishAudio-S1 | 4B | ❌ | 1.72 | 62.57 | 1.22 | 72.1 |
+| Seed-TTS |  | ❌ | 2.25 | 76.2 | 1.12 | 79.6 |
+| MiniMax-Speech |  | ❌ | 1.65 | 69.2 | 0.83 | 78.3 |
+|  |  |  |  |  |  |  |
+| CosyVoice | 0.3B | ✅ | 4.29 | 60.9 | 3.63 | 72.3 |
+| CosyVoice2 | 0.5B | ✅ | 3.09 | 65.9 | 1.38 | 75.7 |
+| CosyVoice3 | 0.5B | ✅ | 2.02 | 71.8 | 1.16 | 78 |
+| CosyVoice3 | 1.5B | ✅ | 2.22 | 72 | 1.12 | 78.1 |
+| F5-TTS | 0.3B | ✅ | 2 | 67 | 1.53 | 76 |
+| SparkTTS | 0.5B | ✅ | 3.14 | 57.3 | 1.54 | 66 |
+| FireRedTTS | 0.5B | ✅ | 3.82 | 46 | 1.51 | 63.5 |
+| FireRedTTS-2 | 1.5B | ✅ | 1.95 | 66.5 | 1.14 | 73.6 |
+| Qwen2.5-Omni | 7B | ✅ | 2.72 | 63.2 | 1.7 | 75.2 |
+| FishAudio-S1-mini | 0.5B | ✅ | 1.94 | 55 | 1.18 | 68.5 |
+| IndexTTS2 | 1.5B | ✅ | 2.23 | 70.6 | 1.03 | 76.5 |
+| VibeVoice | 1.5B | ✅ | 3.04 | 68.9 | 1.16 | 74.4 |
+| HiggsAudio-v2 | 3B | ✅ | 2.44 | 67.7 | 1.5 | 74 |
+| VoxCPM | 0.5B | ✅ | 1.85 | 72.9 | **0.93** | 77.2 |
+| Qwen3-TTS | 0.6B | ✅ | 1.68 | 70.39 | 1.23 | 76.4 |
+| Qwen3-TTS | 1.7B | ✅ | **1.5** | 71.45 | 1.33 | 76.72 |
+|  |  |  |  |  |  |  |
+| MossTTSDelay | 8B | ✅ | 1.79 | 71.46 | 1.32 | 77.05 |
+| MossTTSLocal | 1.7B | ✅ | 1.85 | **73.42** | 1.2 | **78.82** |

docs/moss_tts_realtime_model_card.md

@@ -0,0 +1,213 @@
+# MOSS-TTS-Realtime
+**MOSS-TTS-Realtime** is a context-aware, multi-turn streaming TTS foundation model designed for real-time voice agents. 
+It natively supports spoken interactions by conditioning speech generation on both textual and acoustic history from previous dialogue turns. 
+By tightly integrating multi-turn context modeling with low-latency streaming synthesis, MOSS-TTS-Realtime generates incremental audio responses that preserve voice consistency and discourse coherence, enabling excellent natural and human-like conversational speech.
+
+## 1. Overview
+
+### 1.1 TTS Family Positioning
+
+**MOSS-TTS-Realtime** is a high-performance, real-time speech synthesis model within the broader MOSS TTS Family. It is designed for interactive voice agents that require low-latency, continuous speech generation across multi-turn conversations. Unlike conventional streaming TTS systems that synthesize each response in isolation, MOSS-TTS-Realtime natively models dialogue context by conditioning speech generation on both textual and acoustic information from previous turns. By tightly integrating multi-turn context awareness with incremental streaming synthesis, it produces natural, coherent, and voice-consistent audio responses, enabling fluid and human-like spoken interactions for real-time applications.
+
+**Key Capabilities**
+* **Context-Aware & Expressive Speech Generation**: Generates expressive and coherent speech by modeling both textual and acoustic context across multiple dialogue turns.
+
+* **High-Fidelity Voice Cloning with Multi-Turn Consistency**: Achieves exceptionally high voice similarity while maintaining strong speaker identity consistency across multiple dialogue turns.
+
+* **Long-Context**: Supports long-range context with a maximum context length of 32K (about 40 minutes), enabling stable and consistent speech generation in extended conversations.
+
+* **Highly Human-Like Speech with Natural Prosody**: Trained on over 2.5 million hours of single-speaker speech and more than 1 million hours of two-speaker and multi-speaker conversational data, resulting in highly natural prosody and strong human-like expressiveness.
+
+* **Multilingual Speech Support**: Supports over 10 languages beyond Chinese and English, including Korean, Japanese, German, and French, enabling consistent and expressive speech across languages.
+
+### 1.2 Model Architecture
+
+![MOSS-TTS-Realtime Architecture](../assets/moss_tts_realtime.jpeg)
+
+---
+
+### 1.3 Released Model
+**Recommended decoding hyperparameters**
+| Model | temperature | top_p | top_k | repetition_penalty | repetition_window
+|---|---:|---:|---:|---:|---:|
+| **MOSS-TTS-Realtime** | 0.8 | 0.6 | 30 | 1.1 | 50 |
+
+## 2. Quickstart
+
+### Environment Setup
+Environment setup is the same as on the MOSS-TTS main page.
+
+#### Using Conda
+```bash
+conda create -n moss-tts python=3.12 -y
+conda activate moss-tts
+```
+
+Install all required dependencies:
+
+```bash
+git clone https://github.com/OpenMOSS/MOSS-TTS.git
+cd MOSS-TTS
+pip install --extra-index-url https://download.pytorch.org/whl/cu128 -e .
+cd moss_tts_realtime
+```
+
+#### Using `uv`
+```bash
+# Install uv first: https://docs.astral.sh/uv/getting-started/installation/
+git clone https://github.com/OpenMOSS/MOSS-TTS.git
+cd MOSS-TTS
+uv venv --python 3.12 .venv
+source .venv/bin/activate
+uv pip install --torch-backend cu128 -e .
+cd moss_tts_realtime
+```
+
+### Basic Usage (Non streaming)
+
+```python
+import importlib.util
+import torch
+import torchaudio
+from transformers import AutoTokenizer, AutoModel
+from mossttsrealtime.modeling_mossttsrealtime import MossTTSRealtime
+from inferencer import MossTTSRealtimeInference
+
+CODEC_SAMPLE_RATE = 24000
+
+device = "cuda" if torch.cuda.is_available() else "cpu"
+dtype = torch.bfloat16 if device == "cuda" else torch.float32
+
+def resolve_attn_implementation() -> str:
+    # Prefer FlashAttention 2 when package + device conditions are met.
+    if (
+        device == "cuda"
+        and importlib.util.find_spec("flash_attn") is not None
+        and dtype in {torch.float16, torch.bfloat16}
+    ):
+        major, _ = torch.cuda.get_device_capability()
+        if major >= 8:
+            return "flash_attention_2"
+
+    # CUDA fallback: use PyTorch SDPA kernels.
+    if device == "cuda":
+        return "sdpa"
+
+    # CPU fallback.
+    return "eager"
+
+
+attn_implementation = resolve_attn_implementation()
+print(f"[INFO] Using attn_implementation={attn_implementation}")
+
+model = MossTTSRealtime.from_pretrained("OpenMOSS-Team/MOSS-TTS-Realtime", attn_implementation=attn_implementation, torch_dtype=torch.bfloat16).to(device)
+tokenizer = AutoTokenizer.from_pretrained("OpenMOSS-Team/MOSS-TTS-Realtime")
+codec = AutoModel.from_pretrained("OpenMOSS-Team/MOSS-Audio-Tokenizer", trust_remote_code=True).eval()
+codec = codec.to(device)
+
+inferencer = MossTTSRealtimeInference(model, tokenizer, max_length=5000, codec=codec, codec_sample_rate=CODEC_SAMPLE_RATE, codec_encode_kwargs={"chunk_duration": 8})
+
+text = ["Welcome to the world of MOSS TTS Realtime. Experience how text transforms into smooth, human-like speech in real time.", "MOSS TTS Realtime is a context-aware multi-turn streaming TTS, a speech generation foundation model designed for voice agents."]
+
+# if you don't use reference audio, you can set reference_audio_path = ["", ""]
+reference_audio_path = ["./audio/prompt_audio.mp3", "./audio/prompt_audio1.mp3"]
+
+result = inferencer.generate(
+    text=text,
+    reference_audio_path=reference_audio_path,
+    temperature=0.8,
+    top_p=0.6,
+    top_k=30,
+    repetition_penalty=1.1,
+    repetition_window=50,
+    device=device,
+)
+
+for i, generated_tokens, in enumerate(result):
+    output = torch.tensor(generated_tokens).to(device)
+    decode_result = codec.decode(output.permute(1, 0), chunk_duration=8)
+    wav = decode_result["audio"][0].cpu().detach()
+    torchaudio.save(f'{i}.wav', wav, CODEC_SAMPLE_RATE)
+
+```
+
+### Launch the Gradio streaming demo (recommended)
+You can use streaming output in Gradio with the following usage.
+```bash
+python3 app.py
+```
+
+### Single-turn Streaming Usage
+`example_llm_stream_to_tts.py` demonstrates a single turn has no usage of context:
+```bash
+python3 example_llm_stream_to_tts.py \
+    --model_path OpenMOSS-Team/MOSS-TTS-Realtime \
+    --codec_path OpenMOSS-Team/MOSS-Audio-Tokenizer \
+    --prompt_wav ./audio/prompt_audio1.mp3
+```
+
+Key: provide a streaming text_deltas source that yields incremental text chunks (e.g., vLLM streaming output, or delta text from OpenAI ChatCompletions).
+
+```python
+with codec.streaming(batch_size=1):
+  for delta in text_deltas:
+      print(delta, end="", flush=True)
+      audio_frames = session.push_text(delta)
+      yield from decode_audio_frames(
+          audio_frames, decoder, codebook_size, audio_eos_token
+      )
+
+  audio_frames = session.end_text()
+  yield from decode_audio_frames(
+      audio_frames, decoder, codebook_size, audio_eos_token
+  )
+
+  while True:
+      audio_frames = session.drain(max_steps=1)
+      if not audio_frames:
+          break
+      yield from decode_audio_frames(
+          audio_frames, decoder, codebook_size, audio_eos_token
+      )
+      if session.inferencer.is_finished:
+          break
+
+  yield from flush_decoder(decoder)
+```
+
+### Multi-turn streaming (KV cache reuse)
+
+`example_multiturn_stream_to_tts.py` demonstrates a multi-turn dialogue usage with context:
+- turn 0 resets KV cache
+- turn 1+ reuses KV cache to carry all previous context
+
+```bash
+  python3 example_multiturn_stream_to_tts.py \
+    --model_path OpenMOSS-Team/MOSS-TTS-Realtime \
+    --codec_path OpenMOSS-Team/MOSS-Audio-Tokenizer \
+    --prompt_wav ./audio/prompt_audio1.mp3 
+```
+
+---
+
+
+## 3. Evaluation
+MOSS-TTS-Realtime achieves state-of-the-art or near state-of-the-art performance among open-source systems on the zero-shot TTS benchmarks Seed-TTS-eval, while remaining competitive with leading closed-source models.
+
+| Model | Params | Open-source | EN WER (%) ↓ | EN SIM (%) ↑ | ZH CER (%) ↓ | ZH SIM (%) ↑ |
+|---|---:|:---:|---:|---:|---:|---:|
+| DiTAR | 0.6B | ❌ | 1.69 | 73.5 | 1.02 | 75.3 |
+| CosyVoice3 | 0.5B | ❌ | 2.02 | 71.8 | 1.16 | 78 |
+| CosyVoice3 | 1.5B | ❌ | 2.22 | 72 | 1.12 | 78.1 |
+| FishAudio-S1 | 4B | ❌ | 1.72 | 62.57 | 1.22 | 72.1 |
+| Seed-TTS |  | ❌ | 2.25 | 76.2 | 1.12 | 79.6 |
+| MiniMax-Speech |  | ❌ | 1.65 | 69.2 | 0.83 | 78.3 |
+|  |  |  |  |  |  |  |
+| FishAudio-S1-mini | 0.5B | ✅ | 1.94 | 55 | 1.18 | 68.5 |
+| IndexTTS2 | 1.5B | ✅ | 2.23 | 70.6 | 1.03 | 76.5 |
+| VibeVoice | 1.5B | ✅ | 3.04 | 68.9 | 1.16 | 74.4 |
+| HiggsAudio-v2 | 3B | ✅ | 2.44 | 67.7 | 1.5 | 74 |
+| VoxCPM | 0.5B | ✅ | 1.85 | 72.9 | 0.93 | 77.2 |
+| Qwen3-TTS | 0.6B | ✅ | 1.68 | 70.39 | 1.23 | 76.4 |
+| Qwen3-TTS | 1.7B | ✅ | 1.5 | 71.45 | 1.33 | 76.72 |
+| **Moss-TTS-Realtime** | 1.7B | ✅ | **1.971** | **68.9** | **1.07** | **76.7** |

docs/moss_ttsd_model_card.md

@@ -0,0 +1,250 @@
+# MOSS-TTSD
+
+**MOSS-TTSD** is a long-form spoken dialogue generation model that enables highly expressive multi-party conversational speech synthesis across multiple languages. It supports continuous long-duration generation, flexible multi-speaker dialogue control, and state-of-the-art zero-shot voice cloning with only short reference audio. MOSS-TTSD is designed for real-world long-form content creation, including podcasts, audiobook, sports and esports commentary, dubbing, crosstalk, and entertainment scenarios.
+
+
+## 1. Overview
+
+### 1.1 TTS Family Positioning
+MOSS-TTSD is the Long-Form Dialogue Specialist in our open-source TTS Family. While our foundational models focus on high-fidelity single-speaker synthesis, MOSS-TTSD extends this capability into the realm of complex, multi-party interactions. It is designed to bridge the gap between distinct audio samples and cohesive, continuous conversation.
+
+**Design Goals**
+- **Authentic Interaction**: Capturing the natural rhythm, overlaps, and dynamics of human conversation.
+- **Sustained Coherence**: Maintaining speaker identity and contextual consistency over extended durations (up to 1 hour).
+- **Production Adaptability**: Serving diverse high-end scenarios from rigorous audiobook narration to dynamic sports commentary.
+
+### 1.2 Key Capabilities
+MOSS-TTSD transforms static text into living conversations, offering features specifically optimized for multi-speaker environments:
+
+- **Multi-Party Conversational Generation** — Unlike traditional TTS which optimizes for reading, MOSS-TTSD masters the rhythm of conversation. It supports 1 to 5 speakers with flexible control, handling natural turn-taking, overlapping speech patterns, and distinct persona maintenance.
+
+- **Extreme Long-Context Modeling** — Moving beyond short-sentence generation, the model is architected for stability over long durations, supporting up to 60 minutes of coherent audio in a single session without losing speaker identity or prosodic quality.
+
+- **Diverse Scenario Adaptation** — The model is fine-tuned on high-variability scenarios to handle different speaking styles:
+  - Conversational Media: AI Podcasts, Interviews.
+  - Dynamic Commentary: High-energy Sports/Esports shouting and analysis.
+  - Entertainment: Audiobooks (narrator + characters), Dubbing, and Crosstalk (Xiangsheng).
+
+- **Multilingual & Zero-Shot Cloning** — Features state-of-the-art zero-shot voice cloning requiring only short reference audio (3-10s), with robust cross-lingual performance across major languages including Chinese, English, Japanese, and European languages.
+
+### 1.3 Model Architecture
+
+MOSS-TTSD is built on top of **Delay Pattern (MossTTSDelay)** from our MOSS-TTS foundation model — a single Transformer backbone with multi-head parallel prediction using delay scheduling for multi-codebook audio tokens. 
+
+For full architecture details, see **`moss_tts_delay/README.md`**.
+
+### 1.4 Released Models
+
+| Model | Architecture | NVQ | Parameters |
+|-------|-------------|-----|------------|
+| MOSS-TTSD | Delay Pattern (MossTTSDelay) | 16 | 8B |
+
+**Recommended decoding hyperparameters**
+
+| Model | audio_temperature | audio_top_p | audio_top_k | audio_repetition_penalty |
+|---|---:|---:|---:|---:|
+| **MOSS-TTSD** | 1.1 | 0.9 | 50 | 1.1 |
+
+## 2. Quick Start
+
+MOSS-TTSD uses a **continuation** workflow: provide reference audio for each speaker, their transcripts as a prefix, and the dialogue text to generate. The model continues in each speaker's identity.
+
+```python
+from pathlib import Path
+import importlib.util
+import torch
+import torchaudio
+from transformers import AutoModel, AutoProcessor
+# Disable the broken cuDNN SDPA backend
+torch.backends.cuda.enable_cudnn_sdp(False)
+# Keep these enabled as fallbacks
+torch.backends.cuda.enable_flash_sdp(True)
+torch.backends.cuda.enable_mem_efficient_sdp(True)
+torch.backends.cuda.enable_math_sdp(True)
+
+pretrained_model_name_or_path = "OpenMOSS-Team/MOSS-TTSD-v1.0"
+device = "cuda" if torch.cuda.is_available() else "cpu"
+dtype = torch.bfloat16 if device == "cuda" else torch.float32
+
+def resolve_attn_implementation() -> str:
+    # Prefer FlashAttention 2 when package + device conditions are met.
+    if (
+        device == "cuda"
+        and importlib.util.find_spec("flash_attn") is not None
+        and dtype in {torch.float16, torch.bfloat16}
+    ):
+        major, _ = torch.cuda.get_device_capability()
+        if major >= 8:
+            return "flash_attention_2"
+
+    # CUDA fallback: use PyTorch SDPA kernels.
+    if device == "cuda":
+        return "sdpa"
+
+    # CPU fallback.
+    return "eager"
+
+
+attn_implementation = resolve_attn_implementation()
+print(f"[INFO] Using attn_implementation={attn_implementation}")
+
+processor = AutoProcessor.from_pretrained(
+    pretrained_model_name_or_path,
+    trust_remote_code=True,
+)
+processor.audio_tokenizer = processor.audio_tokenizer.to(device)
+
+model = AutoModel.from_pretrained(
+    pretrained_model_name_or_path,
+    trust_remote_code=True,
+    attn_implementation=attn_implementation,
+    torch_dtype=dtype,
+).to(device)
+model.eval()
+
+# --- Inputs ---
+
+# Use audio from ./assets/audio to avoid downloading from the cloud.
+prompt_audio_speaker1 = "https://speech-demo.oss-cn-shanghai.aliyuncs.com/moss_tts_demo/tts_readme_demo/reference_02_s1.wav"
+prompt_audio_speaker2 = "https://speech-demo.oss-cn-shanghai.aliyuncs.com/moss_tts_demo/tts_readme_demo/reference_02_s2.wav"
+prompt_text_speaker1 = "[S1] In short, we embarked on a mission to make America great again for all Americans."
+prompt_text_speaker2 = "[S2] NVIDIA reinvented computing for the first time after 60 years. In fact, Erwin at IBM knows quite well that the computer has largely been the same since the 60s."
+
+text_to_generate = "[S1] Listen, let's talk business. China. I'm hearing things. People are saying they're catching up. Fast. What's the real scoop? Their AI—is it a threat? [S2] Well, the pace of innovation there is extraordinary, honestly. They have the researchers, and they have the drive. [S1] Extraordinary? I don't like that. I want us to be extraordinary. Are they winning? [S2] I wouldn't say winning, but their progress is very promising. They are building massive clusters. They're very determined. [S1] Promising. There it is. I hate that word. When China is promising, it means we're losing. It's a disaster, Jensen. A total disaster. "
+
+# --- Load & resample audio ---
+
+target_sr = int(processor.model_config.sampling_rate)
+wav1, sr1 = torchaudio.load(prompt_audio_speaker1)
+wav2, sr2 = torchaudio.load(prompt_audio_speaker2)
+
+if wav1.shape[0] > 1:
+    wav1 = wav1.mean(dim=0, keepdim=True)
+if wav2.shape[0] > 1:
+    wav2 = wav2.mean(dim=0, keepdim=True)
+if sr1 != target_sr:
+    wav1 = torchaudio.functional.resample(wav1, sr1, target_sr)
+if sr2 != target_sr:
+    wav2 = torchaudio.functional.resample(wav2, sr2, target_sr)
+
+# --- Build conversation ---
+
+reference_audio_codes = processor.encode_audios_from_wav([wav1, wav2], sampling_rate=target_sr)
+concat_prompt_wav = torch.cat([wav1, wav2], dim=-1)
+prompt_audio = processor.encode_audios_from_wav([concat_prompt_wav], sampling_rate=target_sr)[0]
+
+full_text = f"{prompt_text_speaker1} {prompt_text_speaker2} {text_to_generate}"
+
+conversations = [
+    [
+        processor.build_user_message(
+            text=full_text,
+            reference=reference_audio_codes,
+        ),
+        processor.build_assistant_message(
+            audio_codes_list=[prompt_audio]
+        ),
+    ],
+]
+
+# --- Inference ---
+
+batch_size = 1
+
+save_dir = Path("inference_root")
+save_dir.mkdir(exist_ok=True, parents=True)
+sample_idx = 0
+with torch.no_grad():
+    for start in range(0, len(conversations), batch_size):
+        batch_conversations = conversations[start : start + batch_size]
+        batch = processor(batch_conversations, mode="continuation")
+        input_ids = batch["input_ids"].to(device)
+        attention_mask = batch["attention_mask"].to(device)
+
+        outputs = model.generate(
+            input_ids=input_ids,
+            attention_mask=attention_mask,
+            max_new_tokens=2000,
+        )
+
+        for message in processor.decode(outputs):
+            audio = message.audio_codes_list[0]
+            out_path = save_dir / f"sample{sample_idx}.wav"
+            sample_idx += 1
+            torchaudio.save(out_path, audio.unsqueeze(0), processor.model_config.sampling_rate)
+
+```
+
+### Input Types
+
+**UserMessage**
+
+| Field | Type | Required | Description |
+|---|---|---:|---|
+| `text` | `str` | Yes | Full dialogue text including speaker tags (`[S1]`, `[S2]`, ...) and prompt transcripts. |
+| `reference` | `List` | Yes | Per-speaker reference audio codes from `processor.encode_audios_from_wav()`. |
+
+**AssistantMessage**
+
+| Field | Type | Required | Description |
+|---|---|---:|---|
+| `audio_codes_list` | `List` | Yes | Concatenated prompt audio codes for all speakers. |
+
+### Generation Hyperparameters
+
+| Parameter | Type | Default | Description |
+|---|---|---:|---|
+| `max_new_tokens` | `int` | — | Controls total generated audio tokens. **1s ≈ 12.5 tokens**. |
+| `audio_temperature` | `float` | 1.1 | Higher values increase variation; lower values stabilize prosody. |
+| `audio_top_p` | `float` | 0.9 | Nucleus sampling cutoff. |
+| `audio_top_k` | `int` | 50 | Top-K sampling. |
+| `audio_repetition_penalty` | `float` | 1.1 | >1.0 discourages repeating patterns. |
+
+
+## 3. Evaluation
+### Objective Evaluation(TTSD-eval)
+
+
+
+We introduce a robust evaluation framework leveraging **MMS-FA** for alignment and **wespeaker** for embedding extraction to ensure precise speaker attribution.
+
+
+
+- **Method**: Forced-alignment based segmentation + Similarity-based speaker verification.
+
+- **Metrics**: 
+  - **Speaker Attribution Accuracy (ACC)**
+  - **Speaker Similarity (SIM)**
+  - **Word Error Rate (WER)** computed using **Whisper-large-v3**.
+
+- **Dataset**: 100 multi-turn dialogues (CN/EN) spanning 30s–720s. Covers diverse scenarios including Podcasts, TV dubbing, and Crosstalk. 
+
+Please refer to [TTSD-eval](https://github.com/OpenMOSS/TTSD-eval) for the code and data.
+<br>
+
+| Model | ZH - SIM | ZH - ACC | ZH - WER | EN - SIM | EN - ACC | EN - WER |
+| :--- | :---: | :---: | :---: | :---: | :---: | :---: |
+| **Comparison with Open-Source Models** | | | | | | |
+| MOSS-TTSD | **0.7949** | **0.9587** | **0.0485** | **0.7326** | **0.9626** | 0.0988 |
+| MOSS-TTSD v0.7 | 0.7423 | 0.9391 | 0.0517 | 0.6743 | 0.9266 | 0.1612 |
+| Vibevoice 7B | 0.7590 | 0.9222 | 0.0570 | 0.7140 | 0.9554 | **0.0946** |
+| Vibevoice 1.5 B | 0.7415 | 0.8798 | 0.0818 | 0.6961 | 0.9353 | 0.1133 |
+| FireRedTTS2 | 0.7383 | 0.9022 | 0.0768 | - | - | - |
+| Higgs Audio V2 | - | - | - | 0.6860 | 0.9025 | 0.2131 |
+| **Comparison with Proprietary Models** | | | | | | |
+| Eleven V3 | 0.6970 | 0.9653 | **0.0363** | 0.6730 | 0.9498 | **0.0824** |
+| MOSS-TTSD (elevenlabs_voice) | **0.8165** | **0.9736** | 0.0391 | **0.7304** | **0.9565** | 0.1005 |
+| | | | | | | |
+| gemini-2.5-pro-preview-tts | - | - | - | 0.6786 | 0.9537 | **0.0859** |
+| gemini-2.5-flash-preview-tts | - | - | - | 0.7194 | 0.9511 | 0.0871 |
+| MOSS-TTSD (gemini_voice) | - | - | - | **0.7893** | **0.9655** | 0.0984 |
+| | | | | | | |
+| Doubao_Podcast | 0.8034 | 0.9606 | **0.0472** | - | - | - |
+| MOSS-TTSD (doubao_voice) | **0.8226** | **0.9630** | 0.0571 | - | - | - |
+
+### Subjective Evaluation
+For open-source models, annotators are asked to score each sample pair in terms of speaker attribution accuracy, voice similarity, prosody, and overall quality. Following the methodology of the LMSYS Chatbot Arena, we compute Elo ratings and confidence intervals for each dimension.
+![alt text](../assets/VS_Open-Source_Models.jpg)
+
+For closed-source models, annotators are only asked to choose the overall preferred one in each pair, and we compute the win rate accordingly.
+![alt text](../assets/VS_Proprietary_Models.png)

docs/moss_voice_generator_model_card.md

@@ -0,0 +1,161 @@
+# MOSS-VoiceGenerator Model Card
+
+**MOSS-VoiceGenerator** is an open-source voice generation system designed to enable the creation of custom speaker timbres from free-form textual descriptions. This model allows users to generate voices that reflect specific characters, personalities, and emotions. It is particularly notable for its ability to produce speech with natural-sounding emotional expressiveness, providing a realistic and nuanced listening experience. As an open-source tool, MOSS Voice Generator is suitable for a variety of applications, such as audiobooks, game dubbing, role-playing agents, and conversational assistants.
+
+---
+
+## 1. Overview
+
+### 1.1 TTS Family Positioning
+
+**MOSS-VoiceGenerator** is a high-fidelity voice design tool within the broader TTS Family. It specializes in crafting expressive and natural-sounding voices from textual descriptions. Unlike traditional TTS systems relying on predefined voices or reference audio, MOSS-VoiceGenerator enables zero-shot voice design, allowing for the creation of customized voices for a variety of applications, such as characters, audiobooks, games, or virtual assistants. Additionally, it can serve as a voice design layer for other TTS systems, addressing the challenge of finding suitable reference audio and improving integration and performance.
+
+**Key Capabilities**
+* **Highly expressive emotional delivery**: Aimed at generating voices with dynamic and nuanced emotional performances, allowing for natural shifts in tone, pace, and emotion.
+* **Human-Like Naturalness** : Indistinguishable from real human speech with authentic breathing, pauses, and vocal nuances
+* **Multilingual Support** : High-quality synthesis in Chinese and English
+
+---
+
+### 1.2 Model Architecture
+**MOSS-VoiceGenerator** employs MossTTSDelay architecture (see [moss_tts_delay/README.md](../moss_tts_delay/README.md) for more details), where voice description instructions and the text to be synthesized are concatenated and jointly tokenized as input to drive speech generation, enabling unified modeling of timbre design, style control, and content synthesis. Through instruction-timbre alignment, the model learns the correspondence between textual descriptions and acoustic features, allowing it to generate high-fidelity speech with target timbre, emotion, and style directly from free-form text prompts—without requiring any reference audio. 
+
+### 1.3 Released Model
+**Recommended decoding hyperparameters**
+| Model | audio_temperature | audio_top_p | audio_top_k | audio_repetition_penalty |
+|---|---:|---:|---:|---:|
+| **MOSS-VoiceGenerator** | 1.5 | 0.6 | 50 | 1.1 |
+
+---
+
+## 2. Quick Start
+
+```python
+from pathlib import Path
+import importlib.util
+import torch
+import torchaudio
+from transformers import AutoModel, AutoProcessor
+# Disable the broken cuDNN SDPA backend
+torch.backends.cuda.enable_cudnn_sdp(False)
+# Keep these enabled as fallbacks
+torch.backends.cuda.enable_flash_sdp(True)
+torch.backends.cuda.enable_mem_efficient_sdp(True)
+torch.backends.cuda.enable_math_sdp(True)
+
+pretrained_model_name_or_path = "OpenMOSS-Team/MOSS-VoiceGenerator"
+device = "cuda" if torch.cuda.is_available() else "cpu"
+dtype = torch.bfloat16 if device == "cuda" else torch.float32
+
+def resolve_attn_implementation() -> str:
+    # Prefer FlashAttention 2 when package + device conditions are met.
+    if (
+        device == "cuda"
+        and importlib.util.find_spec("flash_attn") is not None
+        and dtype in {torch.float16, torch.bfloat16}
+    ):
+        major, _ = torch.cuda.get_device_capability()
+        if major >= 8:
+            return "flash_attention_2"
+
+    # CUDA fallback: use PyTorch SDPA kernels.
+    if device == "cuda":
+        return "sdpa"
+
+    # CPU fallback.
+    return "eager"
+
+
+attn_implementation = resolve_attn_implementation()
+print(f"[INFO] Using attn_implementation={attn_implementation}")
+
+processor = AutoProcessor.from_pretrained(
+    pretrained_model_name_or_path,
+    trust_remote_code=True,
+    normalize_inputs=True, # normalize text and instruction input
+)
+processor.audio_tokenizer = processor.audio_tokenizer.to(device)
+
+
+# ====== Batch demo  ======
+text1="哎呀，我的老腰啊，这年纪大了就是不行了。"
+instruction1="疲惫沙哑的老年声音缓慢抱怨，带有轻微呻吟。"
+
+text2="亲爱的观众们，今天我要为大家做一道传说中的龙须面，这道面条细如发丝，需要极其精湛的手艺才能制作成功，请大家仔细观看我的每一个动作。"
+instruction2="热情的美食节目主持人，语调生动活泼，充满对美食的热爱和专业精神。"
+
+text3="Hey there, stranger! What brings you to our humble town? Looking for a good drink or a tall tale?"
+instruction3="Hearty, jovial tavern owner's voice, loud and welcoming with a slightly gruff, friendly tone in American English, radiating warmth and hospitality."
+
+text4="The quick brown fox jumps over the lazy dog."
+instruction4="Clear, neutral voice for phonetic practice, even tempo and precise articulation in standard American English, emphasizing clarity of each word."
+
+conversations = [
+    [processor.build_user_message(text=text1, instruction=instruction1)],
+    [processor.build_user_message(text=text2, instruction=instruction2)],
+    [processor.build_user_message(text=text3, instruction=instruction3)],
+    [processor.build_user_message(text=text4, instruction=instruction4)],
+]
+
+model = AutoModel.from_pretrained(
+    pretrained_model_name_or_path,
+    trust_remote_code=True,
+    attn_implementation=attn_implementation,
+    torch_dtype=dtype,
+).to(device)
+model.eval()
+
+batch_size = 1
+
+save_dir = Path("inference_root")
+save_dir.mkdir(exist_ok=True, parents=True)
+sample_idx = 0
+with torch.no_grad():
+    for start in range(0, len(conversations), batch_size):
+        batch_conversations = conversations[start : start + batch_size]
+        batch = processor(batch_conversations, mode="generation")
+        input_ids = batch["input_ids"].to(device)
+        attention_mask = batch["attention_mask"].to(device)
+
+        outputs = model.generate(
+            input_ids=input_ids,
+            attention_mask=attention_mask,
+        )
+
+        for message in processor.decode(outputs):
+            audio = message.audio_codes_list[0]
+            out_path = save_dir / f"sample{sample_idx}.wav"
+            sample_idx += 1
+            torchaudio.save(out_path, audio.unsqueeze(0), processor.model_config.sampling_rate)
+
+```
+
+### Input Types
+
+**UserMessage**
+
+| Field | Type | Required | Description |
+|---|---|---:|---|
+| `text` | `str` | Yes | Text to synthesize. Supports Chinese and English. |
+| `instruction` | `str` | Yes | Specify the style or the synthesized speech. Users can provide detailed speech style instructions, such as emotion, speed, pitch, and voice characteristics. |
+
+
+### Generation Hyperparameters
+
+| Parameter | Type | Default | Description |
+|---|---|---:|---|
+| `audio_temperature` | `float` | 1.5 | Higher values increase variation; lower values stabilize prosody. |
+| `audio_top_p` | `float` | 0.6 | Nucleus sampling cutoff. Lower values are more conservative. |
+| `audio_top_k` | `int` | 50 | Top-K sampling. Lower values tighten sampling space. |
+| `audio_repetition_penalty` | `float` | 1.1 | >1.0 discourages repeating patterns. |
+
+> Note: MOSS-VoiceGenerator is **sensitive to decoding hyperparameters**. See **Released Models** for recommended defaults.
+
+
+---
+
+
+## 3. Performance
+
+MOSS-VoiceGenerator demonstrates significant advantages in subjective evaluation. Using 160 internal test samples covering diverse voice styles, we established three independent evaluation dimensions: (1) **Overall Preference** - Which voice would you choose? (2) **Instruction Following** - Which audio best follows the instructions (gender, age, tone, emotion, accent, speed)? (3) **Naturalness** - Which audio sounds most like real human speech? Results show that **MOSS-VoiceGenerator outperforms all TTS systems** that support zero predefined voices and customizable preview text across these three dimensions.
+![Win Rate Comparison](../assets/moss_voice_generator_winrate.png)

moss_tts_delay/README.md

@@ -0,0 +1,90 @@
+# Architecture: Global Transformer + Delay-Pattern (MossTTSDelay)
+
+This document details the **MossTTSDelay** architecture, the production-grade variant of the MOSS-TTS family. It employs a **Single Transformer** backbone with **Multi-Head Parallel Prediction** and a **Delay-Pattern** scheduling mechanism to achieve high-speed, stable, and long-form speech synthesis. The architecture diagram is shown in the figure.
+
+<p align="center">
+  <img src="../assets/archi_delay.png" width="60%" />
+</p>
+
+---
+
+## 1. Overview: Parallel Heads + Delay Pattern
+
+Unlike the **MossTTSLocal** architecture which uses a hierarchical "Temporal + Depth" approach, **MossTTSDelay** integrates all modeling into a single large-scale Transformer. It achieves efficient multi-codebook modeling by shifting the RVQ layers in the time domain, allowing the model to predict all codebook layers for a given step simultaneously through multiple linear heads.
+
+
+### Key Components
+*   **Unified Transformer Backbone:** A large-scale language model (based on the **Qwen-8B** scale) that handles text encoding, prosody modeling, and audio token prediction in a single forward pass.
+*   **Multi-Head Output Layer:** The backbone is equipped with **$1 + N_q$** (where $N_q=32$) prediction heads. One head manages the primary sequence logic, while the other 32 heads parallelly predict the RVQ codebook layers.
+*   **Delay-Pattern Scheduling:** A specialized data formatting technique that introduces a 1-step offset between successive RVQ layers. This enables causal dependency modeling across codebook depths without requiring an additional "Depth Transformer."
+
+---
+
+## 2. Technical Specifications
+
+| Feature | Specification |
+| :--- | :--- |
+| **Backbone Model** | Initialized from **Qwen-8B** scale |
+| **Prediction Heads** | **33 LM Heads** (1 Main + 32 RVQ Heads) |
+| **Audio Tokenizer** | **Cat** (Causal Audio Tokenizer) |
+| **Sampling Rate** | 24,000 Hz |
+| **Frame Rate** | 12.5 Hz (1s ≈ 12.5 tokens) |
+| **Codebooks** | 32 RVQ layers (10-bit each) |
+| **Generation Mode** | Parallel Autoregressive (Delay-Pattern) |
+| **Primary Advantage** | Inference speed & Long-context stability |
+
+---
+
+## 3. Core Mechanism: Multi-Head Parallel Prediction
+
+The defining characteristic of MossTTSDelay is its **computational efficiency**. By attaching 32 independent linear heads to the final hidden state of the Transformer backbone, the model can generate an entire frame's worth of multi-layer RVQ tokens in a **single forward step**.
+
+### Why this is faster than MossTTSLocal:
+*   **No Nested Loops:** While the Local architecture requires a secondary "Local Transformer" to iterate through each RVQ layer within one time step, MossTTSDelay computes all layers in parallel.
+*   **Direct Projection:** The relationship between codebook layers is captured by the backbone's internal representations and the delay-pattern, removing the latency overhead of a dedicated depth-modeling module.
+
+---
+
+## 4. Prediction Topology: Delay-Pattern
+
+To maintain the hierarchical dependency of RVQ (where Layer $k$ should ideally "see" the information from Layer $k-1$), MossTTSDelay uses **Delay-Pattern Scheduling**. 
+
+**The Pattern:**
+At each training or inference step $t$, the input sequence is structured such that:
+*   Head 1 predicts Layer 1 of Frame $t$.
+*   Head 2 predicts Layer 2 of Frame $t-1$.
+*   Head 3 predicts Layer 3 of Frame $t-2$.
+*   ... and so on.
+
+**Dependency Modeling:**
+Because the Transformer is causal, when the model predicts tokens for "Step $t$", it has already seen the tokens from "Step $t-1$" in its context. Due to the 1-step shift, the information for Layer $k-1$ (at Step $t$) is already present in the history when the model predicts Layer $k$ (at Step $t+1$). This "diagonal" dependency effectively models the coarse-to-fine structure of the audio tokenizer.
+
+---
+
+## 5. Evaluation & Performance
+
+According to the `moss_tts_model_card.md`, the **MossTTSDelay-8B** is the recommended model for production and long-form stability:
+
+| Metric | Result (Seed-TTS-Eval) |
+| :--- | :--- |
+| **EN SIM (Speaker Similarity)** | **0.7146** |
+| **ZH SIM (Speaker Similarity)** | **0.7705** |
+| **EN WER (Word Error Rate)** | **1.79%** |
+| **ZH CER (Char Error Rate)** | **1.32%** |
+
+**Conclusion:** MossTTSDelay offers superior long-context stability and faster inference speeds compared to the Local variant. Its 8B parameter scale provides the capacity needed for complex prosody and ultra-long (up to 1 hour) speech generation.
+
+---
+
+## 6. Architecture Comparison
+
+| Aspect | MossTTSDelay (Architecture A) | MossTTSLocal (Architecture B) |
+| :--- | :--- | :--- |
+| **Structure** | Single Transformer (8B) | Temporal + Depth Transformers (1.7B) |
+| **Scheduling** | **Delay-Pattern (Diagonal Shift)** | Per-step Synchronous Blocks |
+| **Prediction Heads** | **33 Parallel Heads** | Single Latent Head + Local Module |
+| **Inference Speed** | **High** (Parallel RVQ prediction) | Moderate (Sequential RVQ prediction) |
+| **Stability** | Excellent for long-form (1h+) | Optimized for short-segment metrics |
+| **Best For** | Production, Scalable Apps, Narration | Research, Quality Benchmarks |
+
+---