rwkv-x-dev

community

AI & ML interests

hugging face space, for rwkv-x related developments, including build assets, etc Nothing in here are considered "official releases" AKA - we treat this as a giant file dump

SmerkyG

authored a paper 11 months ago

RADLADS: Rapid Attention Distillation to Linear Attention Decoders at Scale

Paper • 2505.03005 • Published May 5, 2025 • 36

m8than

authored a paper about 1 year ago

RWKV-7 "Goose" with Expressive Dynamic State Evolution

Paper • 2503.14456 • Published Mar 18, 2025 • 154

SmerkyG

authored a paper about 1 year ago

Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence

Paper • 2404.05892 • Published Apr 8, 2024 • 40

BlinkDL

authored a paper about 1 year ago

Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence

Paper • 2404.05892 • Published Apr 8, 2024 • 40

SmerkyG

authored 2 papers about 1 year ago

GoldFinch: High Performance RWKV/Transformer Hybrid with Linear Pre-Fill and Extreme KV-Cache Compression

Paper • 2407.12077 • Published Jul 16, 2024 • 57

RWKV-7 "Goose" with Expressive Dynamic State Evolution

Paper • 2503.14456 • Published Mar 18, 2025 • 154

ZhangRC

authored a paper about 1 year ago

RWKV-7 "Goose" with Expressive Dynamic State Evolution

Paper • 2503.14456 • Published Mar 18, 2025 • 154

BlinkDL

authored a paper about 1 year ago

RWKV-7 "Goose" with Expressive Dynamic State Evolution

Paper • 2503.14456 • Published Mar 18, 2025 • 154

ZhangRC

in rwkv-x-dev/rwkv-world-3-subsample-preview about 1 year ago

[bot] Conversion to Parquet

#1 opened about 1 year ago by

parquet-converter

ZhangRC

updated a dataset about 1 year ago

rwkv-x-dev/rwkv-world-3-subsample-preview

Viewer • Updated Mar 16, 2025 • 1M • 15 • 2

ZhangRC

published a dataset about 1 year ago

rwkv-x-dev/rwkv-world-3-subsample-preview

Viewer • Updated Mar 16, 2025 • 1M • 15 • 2

BlinkDL

posted an update over 1 year ago

Post

13155

RWKV-7 "Goose" 0.4B trained w/ ctx4k automatically extrapolates to ctx32k+, and perfectly solves NIAH ctx16k 🤯 100% RNN and attention-free. Only trained on the Pile. No finetuning. Replicable training runs. tested by our community: https://github.com/Jellyfish042/LongMamba

Hazzzardous

authored a paper over 1 year ago

Lina-Speech: Gated Linear Attention is a Fast and Parameter-Efficient Learner for text-to-speech synthesis

Paper • 2410.23320 • Published Oct 30, 2024 • 8

BlinkDL

posted an update over 1 year ago

Post

13783

RWKV-6-world-v3 (+3.1T tokens) is our best multilingual 7B model as of now: BlinkDL/rwkv-6-world

It's 100% RNN and attention-free. MMLU 54.2% (previous world-v2.1 = 47.9%. note: without eval-boosting tricks such as annealing).

RWKV-7-world-v4 soon :)

1 reply

BlinkDL

posted an update over 1 year ago

Post

5832

RWKV-7 "Goose" preview rc2 => Peak RNN architecture?😃Will try to squeeze more performance for the final release. Preview code & model: https://github.com/BlinkDL/RWKV-LM/tree/main/RWKV-v7