haval995/roberta-large-causal-hallucination-detector

RoBERTa-Large Causal Relation Detector

A fine-tuned RoBERTa-large model for detecting causal relationships in text. Classifies sentences as causal (1) or non-causal (0). Designed as the causal claim extraction component of a Causal Hallucination Detection Pipeline for Retrieval-Augmented Generation (RAG) systems.

Model Description

• Model type: Text Classification (Binary)
• Language: English
• License: Apache 2.0
• Base model: roberta-large
• Developed by: Haval
• Fine-tuned for: Causal relation detection in the context of RAG hallucination detection

What It Does

Given a sentence, the model predicts whether it contains a causal relationship:

• Label 1 (Causal): "Smoking causes lung cancer because it damages lung cells."
• Label 0 (Non-Causal): "The conference was held in Berlin last year."

How to Use

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

model_name = "your-username/roberta-large-causal-hallucination-detector"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)
model.eval()

def predict_causal(sentence):
    inputs = tokenizer(
        sentence,
        return_tensors="pt",
        truncation=True,
        max_length=128
    )
    with torch.no_grad():
        outputs = model(**inputs)
        probs = torch.softmax(outputs.logits, dim=-1)
        label = torch.argmax(probs).item()
        confidence = probs[0][label].item()
    return {
        "label": "causal" if label == 1 else "non-causal",
        "confidence": round(confidence, 4)
    }

print(predict_causal("Smoking causes lung cancer because it damages lung cells."))
# Output: {"label": "causal", "confidence": 0.97}

print(predict_causal("The meeting was scheduled for Monday morning."))
# Output: {"label": "non-causal", "confidence": 0.95}

Training Data

Trained on a combined and balanced dataset from two sources:

Dataset	Examples	Causal	Non-Causal
SemEval-2010 Task 8	8,000	1,003	6,997
BECAUSE 2.0	1,231	403	828
Combined & Balanced	2,812	1,406	1,406

Split: 1,968 train / 422 validation / 422 test (70/15/15)

Causal relation types from BECAUSE 2.0: Motivation, Consequence, Purpose, Cause-Effect

Causal relation types from SemEval-2010: Cause-Effect(e1,e2), Cause-Effect(e2,e1)

Training Hyperparameters

Parameter	Value
Base model	roberta-large
Epochs	5
Batch size	8
Learning rate	1e-5
Warmup steps	100
Weight decay	0.01
Max sequence length	128
Optimizer	AdamW
Hardware	Google Colab T4 GPU
Training time	~30 minutes

Evaluation Results

Evaluated on a held-out test set of 422 examples (211 causal, 211 non-causal):

Metric	Score
Accuracy	92.4%
F1	0.928
Precision	0.919
Recall	0.936

Per-Class Results

Class	Precision	Recall	F1	Support
Non-Causal	0.93	0.91	0.92	203
Causal	0.92	0.94	0.93	219

Comparison with Baseline

Model	F1	Accuracy
roberta-base (baseline)	0.909	90.3%
roberta-large (this model)	0.928	92.4%

Intended Use

This model is designed to be used as the first component in a causal hallucination detection pipeline for RAG systems. It identifies which sentences contain causal claims that need to be verified against source documents.

Intended users: NLP researchers, RAG system developers, hallucination detection systems

Out-of-Scope Use

• Not designed for detecting non-causal hallucinations
• Not tested on languages other than English
• Not suitable for detecting cross-sentence causal relationships (inter-sentence causality)
• Should not be used as a standalone hallucination detector without the full pipeline

Limitations

• Detects causal relationships at the sentence level only — cross-sentence causal relationships are not captured
• Trained on general and news domain data — performance may vary on highly technical domains (medical, legal) without domain-specific fine-tuning
• English only

Part of a Larger Pipeline

This model is the causal extraction component of a 4-step hallucination detection pipeline:

Step 1: Causal Claim Extraction    ← This model
Step 2: Document Grounding         (SBERT + Cross-encoder + Gemini)
Step 3: NLI Entailment Checking    (DeBERTa-v3-large-mnli)
Step 4: LLM-as-Judge Scoring       (Gemini 1.5 Flash)

Citation

If you use this model in your research, please cite:

@misc{roberta-causal-hallucination-detector,
  author = {Haval},
  title = {RoBERTa-Large Causal Relation Detector for RAG Hallucination Detection},
  year = {2025},
  publisher = {HuggingFace},
  url = {https://huggingface.co/your-username/roberta-large-causal-hallucination-detector}
}

Environmental Impact

• Hardware: Google Colab T4 GPU
• Training time: ~30 minutes
• Cloud provider: Google Colab