RobAgrees/distilhubert-finetuned-gtzan
23.7M
•
Updated
Robert Agee
RobAgrees
AI & ML interests
None yet
Recent Activity
updated
a model
about 7 hours ago
RobAgrees/distilhubert-finetuned-gtzan
published
a model
about 8 hours ago
RobAgrees/distilhubert-finetuned-gtzan
liked
a model
about 1 month ago
ReadyArt/4.2.0-Broken-Tutu-24b
Organizations
None yet
updated
a
model
about 7 hours ago
published
a
model
about 8 hours ago
liked
a
model
about 1 month ago
This model is special
❤️
2
1
#1 opened about 1 month ago
by
RobAgrees
72GB VRAM Users - What size quants do you want (70B models) ?
6
#2 opened 5 months ago
by
FrenzyBiscuit
liked
a
model
4 months ago
[Feedback] Hardware Compatibility for AMD Ryzen AI Max+ 395 aka Strix Halo
👍
20
#73 opened 5 months ago
by
RobAgrees
liked
a
Space
6 months ago
reacted to
ProCreations's
post with 🚀
7 months ago
Post
2940
Eyyyy 50 followers 🤯
- 1 reply
Driver Code or GemmaCPP support?
🤗
8
4
#7 opened 7 months ago
by
yoyou446
Adding Evaluation Results
1
#4 opened about 1 year ago
by
leaderboard-pr-bot
This thing is hardly evil at all
#1 opened 7 months ago
by
RobAgrees
reacted to
codys12's
post with 🚀👀
7 months ago
Post
2325
Introducing bitnet-r1-llama-8b and bitnet-r1-qwen-32b preview! These models are the first successful sub 1-billion-token finetune to BitNet architecture. We discovered that by adding an aditional input RMSNorm to each linear, you can finetune directly to BitNet with fast convergence to original model performance!
We are working on a pull request to use this extra RMS for any model.
To test these models now, install this fork of transformers:
Then load the models and test:
bitnet-r1-llama-8b and bitnet-r1-llama-32b were trained on ~ 300M and 200M tokens of the open-thoughts/OpenThoughts-114k dataset respectively, and were still significantly improving at the end of training. This preview simply demonstrates that the concept works, for future training runs we will leave the lm_head unquantized and align the last hidden state with the original model.
Huge thanks to the team that made this possible:
Gavin Childress, Aaron Herbst, Gavin Jones, Jasdeep Singh, Eli Vang, and Keagan Weinstock from the MSOE AI Club.
We are working on a pull request to use this extra RMS for any model.
To test these models now, install this fork of transformers:
pip install git+https://github.com/Codys12/transformers.gitThen load the models and test:
from transformers import (AutoModelForCausalLM, AutoTokenizer)
model_id = "codys12/bitnet-r1-qwen-32b"
model = AutoModelForCausalLM.from_pretrained(
model_id,
device_map="cuda",
)
tokenizer = AutoTokenizer.from_pretrained(model_id, padding_side="left")bitnet-r1-llama-8b and bitnet-r1-llama-32b were trained on ~ 300M and 200M tokens of the open-thoughts/OpenThoughts-114k dataset respectively, and were still significantly improving at the end of training. This preview simply demonstrates that the concept works, for future training runs we will leave the lm_head unquantized and align the last hidden state with the original model.
Huge thanks to the team that made this possible:
Gavin Childress, Aaron Herbst, Gavin Jones, Jasdeep Singh, Eli Vang, and Keagan Weinstock from the MSOE AI Club.
- 1 reply
updated
a
model
8 months ago
published
a
model
8 months ago
when it will be available for open source community
➕
3
17
#1 opened about 1 year ago
by
arpitsh018
liked
a
model
12 months ago
liked
a
Space
12 months ago
liked
a
Space
over 1 year ago