29m10/distilbert-base-uncased-finetuned-intent

Model Card for DistilBERT Intent Classification with LoRA

This model is a fine-tuned version of distilbert-base-uncased for intent classification using LoRA (Low-Rank Adaptation) on the Amazon MASSIVE Intent dataset. The model can classify user intents across 60 different categories with 87.79% accuracy.

Model Details

Model Description

This is a DistilBERT-based sequence classification model fine-tuned for intent recognition using Parameter-Efficient Fine-Tuning (PEFT) with LoRA adapters. The model achieves strong performance on intent classification while being computationally efficient, requiring only ~3.14MB for the LoRA adapter weights.

• Developed by: 29m10
• Shared by: 29m10
• Model type: Text Classification (Intent Recognition)
• Language(s) (NLP): English
• License: Apache 2.0
• Finetuned from model: distilbert-base-uncased

Model Sources

• Repository: 29m10/distilbert-base-uncased-finetuned-intent

Uses

Direct Use

The model can be used directly for intent classification in conversational AI systems, chatbots, or voice assistants to understand user intents from natural language input. It classifies text into 60 different intent categories from the Amazon MASSIVE dataset.

Downstream Use

This model can be integrated into larger conversational AI pipelines, customer service automation systems, or used as a component in multi-intent classification systems. It can also be further fine-tuned for domain-specific intent classification tasks.

Out-of-Scope Use

This model is specifically trained for English intent classification on the MASSIVE dataset domains. It may not perform well on non-English text, intents outside the 60 categories it was trained on, or domain-specific intents not covered in the training data.

Bias, Risks, and Limitations

The model inherits biases present in the Amazon MASSIVE Intent dataset. Performance may vary across different demographic groups or linguistic variations. The model is limited to the 60 intent categories present in the training data and may not generalize well to out-of-domain intents.

Recommendations

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. For production systems, implement appropriate fallback mechanisms for out-of-scope intents. Evaluate the model's performance on your specific use case and consider potential biases.

How to Get Started with the Model

Use the code below to get started with the model.

from transformers import AutoTokenizer, AutoModelForSequenceClassification
from peft import PeftModel

# Load tokenizer and base model
tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")
base_model = AutoModelForSequenceClassification.from_pretrained("distilbert-base-uncased", num_labels=60)

# Load LoRA adapter
model = PeftModel.from_pretrained(base_model, "29m10/distilbert-base-uncased-finetuned-intent")

# Use with pipeline
from transformers import pipeline
classifier = pipeline("text-classification", model=model, tokenizer=tokenizer, top_k=1)

# Example usage
result = classifier("please cancel my alarm")
print(result)
# Output: [{'label': 'alarm_remove', 'score': 0.9957757592201233}]

Training Details

Training Data

The model was trained on the English subset of the Amazon MASSIVE Intent dataset, which contains multilingual training data for intent classification with 60 different intent categories including alarm management, weather queries, music control, and other common voice assistant tasks.

Training Procedure

Preprocessing

Text inputs were tokenized using the DistilBERT tokenizer with a maximum sequence length of 128 tokens. Padding was applied to reach the maximum length, and string labels were converted to numerical IDs for the 60 intent categories.

Training Hyperparameters

• Training regime: LoRA fine-tuning with fp32 precision
• LoRA Configuration:
  • LoRA rank (r): 8
  • LoRA alpha: 16
  • LoRA dropout: 0.05
  • Target modules: ["q_lin", "v_lin"] (attention projection layers)
  • Bias: none
• Learning rate: 2e-4
• Batch size: 32 (both train and eval)
• Number of epochs: 10
• Weight decay: 0.01
• LR scheduler: Linear with 5% warmup ratio
• Optimizer: AdamW (default)
• Evaluation strategy: Every epoch
• Save strategy: Every epoch
• Metric for best model: F1-macro

Speeds, Sizes, Times

• Model size: ~3.14MB (LoRA adapter only)
• Training epochs: 10
• Evaluation speed: ~274 samples/second, ~8.6 steps/second
• Hardware: NVIDIA T4 GPU

Evaluation

Testing Data, Factors & Metrics

Testing Data

The model was evaluated on the test split of the Amazon MASSIVE Intent dataset (English subset).

Factors

Evaluation was performed across all 60 intent categories without disaggregation by subpopulations or specific domains.

Metrics

• Accuracy: Overall classification accuracy across all intent categories
• F1-macro: Macro-averaged F1 score to account for potential class imbalance across the 60 intent categories

Results

Final Test Results:

• Accuracy: 87.79%
• F1-macro: 86.28%
• Test Loss: 0.453
• Evaluation runtime: 10.8449 seconds

Summary

The model demonstrates strong performance across the 60 intent categories with well-balanced precision and recall as indicated by the close accuracy and F1-macro scores.

Environmental Impact

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019). The use of LoRA significantly reduces computational requirements compared to full fine-tuning, resulting in lower energy consumption.

• Hardware Type: NVIDIA T4 GPU
• Cloud Provider: Google Colab

Technical Specifications

Model Architecture and Objective

• Base Architecture: DistilBERT (6 transformer layers, 768 hidden size)
• Fine-tuning Method: LoRA (Low-Rank Adaptation)
• Objective: Multi-class classification with cross-entropy loss
• Number of Classes: 60 intent categories

Compute Infrastructure

Hardware

• GPU: NVIDIA T4
• Environment: Google Colab

Software

• Framework: PyTorch 2.8.0+cu126
• Libraries: transformers 4.57.1, peft 0.17.1, datasets 4.4.1, accelerate 1.11.0

Citation

BibTeX:

@misc{distilbert-intent-lora-2024,
  title={DistilBERT Intent Classification with LoRA},
  author={29m10},
  year={2024},
  howpublished={Hugging Face Model Hub},
  url={https://huggingface.co/29m10/distilbert-base-uncased-finetuned-intent}
}

APA:

29m10. (2024). DistilBERT Intent Classification with LoRA. Hugging Face Model Hub. https://huggingface.co/29m10/distilbert-base-uncased-finetuned-intent

Glossary

• LoRA (Low-Rank Adaptation): A parameter-efficient fine-tuning technique that adds trainable low-rank matrices to existing weights
• Intent Classification: The task of determining the intent or purpose behind a user's natural language input
• PEFT: Parameter-Efficient Fine-Tuning methods that update only a small subset of model parameters

More Information

For more details about the LoRA technique, see the original paper: LoRA: Low-Rank Adaptation of Large Language Models (https://arxiv.org/abs/2106.09685)

Model Card Authors

29m10

Framework versions

• PEFT 0.17.1
• Transformers 4.57.1
• PyTorch 2.8.0+cu126
• Datasets 4.4.1
• Accelerate 1.11.0