|
|
--- |
|
|
datasets: |
|
|
- GetSoloTech/FoodStack |
|
|
language: |
|
|
- en |
|
|
base_model: |
|
|
- lerobot/smolvla_base |
|
|
library_name: transformers |
|
|
tags: |
|
|
- Robotics |
|
|
- Lerobot |
|
|
- Food |
|
|
- PickPlace |
|
|
- VLA |
|
|
- SmolVLA |
|
|
- PhysicalAI |
|
|
--- |
|
|
|
|
|
### SmolVLA Fine-Tuned on for Food Stacking |
|
|
|
|
|
**Summary**: This is a fine-tuned version of `lerobot/smolvla_base` for stacking food objects (e.g., burgers, sandwiches). It was fine-tuned on the `GetSoloTech/FoodStack` dataset using the LeRobot framework. |
|
|
|
|
|
### Model details |
|
|
- **Base model**: `lerobot/smolvla_base` |
|
|
- **Task**: Vision-Language-Action control for manipulation (stacking) |
|
|
- **Domain**: Food item stacking (burger, sandwich, etc.) |
|
|
- **Params**: ~450M (SmolVLA) |
|
|
- **Library**: LeRobot (`lerobot`) |
|
|
|
|
|
### Quick start |
|
|
Install LeRobot with SmolVLA extras: |
|
|
|
|
|
```bash |
|
|
git clone https://github.com/huggingface/lerobot.git |
|
|
cd lerobot |
|
|
pip install -e ".[smolvla]" |
|
|
``` |
|
|
|
|
|
Load the policy from this repo and run inference: |
|
|
|
|
|
```python |
|
|
from lerobot.common.policies.smolvla.modeling_smolvla import SmolVLAPolicy |
|
|
|
|
|
# Replace with your actual model ID on the Hub |
|
|
model_id = "GetSoloTech/SmolVLA-FoodStack" |
|
|
|
|
|
policy = SmolVLAPolicy.from_pretrained(model_id) |
|
|
|
|
|
# Example placeholders for observation and instruction |
|
|
observation = { |
|
|
"image": ... , # BGR/RGB frame or processed observation per your setup |
|
|
"state": ... , # optional proprio/scene state if used |
|
|
} |
|
|
instruction = "Stack the burger: bun, patty, cheese, lettuce, bun." |
|
|
|
|
|
# Depending on your pipeline, you may wrap this in your control loop |
|
|
actions = policy(observation, instruction) |
|
|
|
|
|
# Send actions to your robot controller |
|
|
# send_actions_to_robot(actions) |
|
|
``` |
|
|
|
|
|
For end-to-end examples (policy loops, camera/robot IO), see the LeRobot docs and examples. |
|
|
|
|
|
|
|
|
Notes: |
|
|
- Tune batch size/steps and augmentation to your hardware and dataset split. |
|
|
- Ensure your observation preprocessing at train-time matches inference. |
|
|
|
|
|
|
|
|
### Limitations |
|
|
- Specializes in food stacking; may not generalize to unseen objects/layouts. |
|
|
- Sensitive to perception domain shift (lighting, textures, camera intrinsics). |
|
|
- Requires correct observation normalization consistent with training. |
|
|
|
|
|
### Dataset |
|
|
- **Training data**: `GetSoloTech/FoodStack` |
|
|
|
|
|
### Resources and references |
|
|
- SmolVLA base: `https://huggingface.co/lerobot/smolvla_base` |
|
|
- SmolVLA overview: `https://smolvla.net/index_en.html` |
|
|
- LeRobot: `https://github.com/huggingface/lerobot` |
