Added Model Card

README.md

@@ -0,0 +1,208 @@
+---
+datasets:
+- ExponentialScience/DLT-Tweets
+- ExponentialScience/DLT-Patents
+- ExponentialScience/DLT-Scientific-Literature
+language:
+- en
+base_model:
+- allenai/scibert_scivocab_cased
+---
+# LedgerBERT
+
+## Model Description
+
+### Model Summary
+
+LedgerBERT is a domain-adapted language model specialized for the Distributed Ledger Technology (DLT) field. It was created through continual pre-training of SciBERT on the DLT-Corpus, a comprehensive collection of 2.98 billion tokens from scientific literature, patents, and social media focused on blockchain, cryptocurrencies, and distributed ledger systems.
+
+LedgerBERT captures DLT-specific terminology and concepts, making it particularly effective for NLP tasks involving blockchain technologies, cryptocurrency discourse, smart contracts, consensus mechanisms, and related domain-specific content.
+
+- **Developed by:** Walter Hernandez Cruz, Peter Devine, Nikhil Vadgama, Paolo Tasca, Jiahua Xu
+- **Model type:** BERT-base encoder (bidirectional transformer)
+- **Language:** English
+- **License:** CC-BY-NC 4.0 (Creative Commons Attribution-NonCommercial 4.0 International)
+- **Base model:** SciBERT (allenai/scibert_scivocab_cased)
+- **Training corpus:** DLT-Corpus (2.98 billion tokens)
+
+### Model Architecture
+
+- **Architecture:** BERT-base
+- **Parameters:** 110 million
+- **Hidden size:** 768
+- **Number of layers:** 12
+- **Attention heads:** 12
+- **Vocabulary size:** 30,522 (SciBERT vocabulary)
+- **Max sequence length:** 512 tokens
+
+## Intended Uses
+
+### Primary Use Cases
+
+LedgerBERT is designed for NLP tasks in the DLT domain, including, but not limited to:
+
+- **Named Entity Recognition (NER)**: Identifying DLT-specific entities such as consensus mechanisms (e.g., Proof of Stake), blockchain platforms (e.g., Ethereum, Hedera), cryptographic concepts (e.g., Merkle tree, hashing)
+- **Text Classification**: Categorizing DLT-related documents, patents, or social media posts
+- **Sentiment Analysis**: Analyzing sentiment in cryptocurrency news and social media
+- **Information Extraction**: Extracting technical concepts and relationships from DLT literature
+- **Document Retrieval**: Building search systems for DLT content
+- **Question Answering (QA)**: Creating QA systems for blockchain and cryptocurrency topics
+
+### Out-of-Scope Uses
+
+- **Real-time trading systems**: LedgerBERT should not be used as the sole basis for automated trading decisions
+- **Investment advice**: Not suitable for providing financial or investment recommendations without proper disclaimers
+- **General-purpose NLP**: While LedgerBERT maintains general language understanding, it is optimized for DLT-specific tasks
+- **Legal or regulatory compliance**: Should not be used for legal interpretation without expert review
+
+## Training Details
+
+### Training Data
+
+LedgerBERT was continually pre-trained on the **DLT-Corpus**, consisting of:
+
+- **Scientific Literature**: 37,440 documents, 564M tokens (1978-2025). See https://huggingface.co/datasets/ExponentialScience/DLT-Scientific-Literature
+- **Patents**: 49,023 documents, 1,296M tokens (1990-2025). See https://huggingface.co/datasets/ExponentialScience/DLT-Patents
+- **Social Media**: 22.03M documents, 1,120M tokens (2013-mid 2023). See https://huggingface.co/datasets/ExponentialScience/DLT-Tweets
+
+**Total:** 22.12 million documents, 2.98 billion tokens
+
+For more details, see: https://huggingface.co/collections/ExponentialScience/dlt-corpus-68e44e40d4e7a3bd7a224402
+
+### Training Procedure
+
+**Continual Pre-training:**
+
+Starting from SciBERT (which already captures multidisciplinary scientific content), LedgerBERT was trained using Masked Language Modeling (MLM) on the DLT-Corpus to adapt the model to DLT-specific terminology and concepts.
+
+**Training hyperparameters:**
+- **Epochs:** 3
+- **Learning rate:** 5×10⁻⁵ with linear decay schedule
+- **MLM probability:** 0.15 (standard BERT masking)
+- **Warmup ratio:** 0.10
+- **Batch size:** 12 per device
+- **Sequence length:** 512 tokens
+- **Weight decay:** 0.01
+- **Optimizer:** Stable AdamW
+- **Precision:** bfloat16
+
+
+## Limitations and Biases
+
+### Known Limitations
+
+- **Language coverage**: English only; does not support other languages
+- **Temporal coverage**: Training data extends to mid-2023 for social media; may not capture very recent terminology
+- **Domain specificity**: Optimized for DLT tasks; may underperform on general-purpose benchmarks compared to models like RoBERTa
+- **Context length**: Limited to 512 tokens; longer documents require truncation or chunking
+
+### Potential Biases
+
+The model may reflect biases present in the training data:
+
+- **Geographic bias**: English-language sources may over-represent certain regions
+- **Platform bias**: Social media data only from Twitter/X; other platforms not represented
+- **Temporal bias**: More recent DLT developments are more heavily represented
+- **Market bias**: Training during periods of market volatility may influence sentiment understanding
+- **Source bias**: Certain cryptocurrencies (e.g., Bitcoin, Ethereum) are more discussed than others
+
+### Ethical Considerations
+
+- **Market manipulation risk**: Could potentially be misused for analyzing or generating content for market manipulation
+- **Investment decisions**: Should not be used as sole basis for financial decisions without proper risk disclaimers
+- **Misinformation**: May reproduce or fail to identify false claims present in training data
+- **Privacy**: While usernames were removed from social media data, care should be taken not to re-identify individuals
+
+## How to Use
+
+### Basic Usage
+
+```python
+from transformers import AutoTokenizer, AutoModel
+
+# Load model and tokenizer
+tokenizer = AutoTokenizer.from_pretrained("ExponentialScience/LedgerBERT")
+model = AutoModel.from_pretrained("ExponentialScience/LedgerBERT")
+
+# Example text
+text = "Ethereum uses Proof of Stake consensus mechanism for transaction validation."
+
+# Tokenize and encode
+inputs = tokenizer(text, return_tensors="pt", max_length=512, truncation=True)
+
+# Get embeddings
+outputs = model(**inputs)
+embeddings = outputs.last_hidden_state
+```
+
+### Fine-tuning for NER
+
+```python
+from transformers import AutoTokenizer, AutoModelForTokenClassification, TrainingArguments, Trainer
+
+# Load for token classification
+tokenizer = AutoTokenizer.from_pretrained("ExponentialScience/LedgerBERT")
+model = AutoModelForTokenClassification.from_pretrained(
+    "ExponentialScience/LedgerBERT",
+    num_labels=num_labels  # Set based on your NER task
+)
+
+# Fine-tune on your dataset
+training_args = TrainingArguments(
+    output_dir="./results",
+    learning_rate=1e-5,
+    per_device_train_batch_size=16,
+    num_train_epochs=20,
+    warmup_steps=500
+)
+
+trainer = Trainer(
+    model=model,
+    args=training_args,
+    train_dataset=train_dataset,
+    eval_dataset=eval_dataset
+)
+
+trainer.train()
+```
+
+### Fine-tuning for Sentiment Analysis
+
+A fine-tuned version for market sentiment is available at: https://huggingface.co/ExponentialScience/LedgerBERT-Market-Sentiment
+
+```python
+from transformers import AutoTokenizer, AutoModelForSequenceClassification
+
+tokenizer = AutoTokenizer.from_pretrained("ExponentialScience/LedgerBERT-Market-Sentiment")
+model = AutoModelForSequenceClassification.from_pretrained("ExponentialScience/LedgerBERT-Market-Sentiment")
+
+text = "Bitcoin reaches new all-time high amid institutional adoption"
+inputs = tokenizer(text, return_tensors="pt", max_length=512, truncation=True)
+outputs = model(**inputs)
+predictions = outputs.logits.argmax(dim=-1)
+```
+
+## Citation
+
+If you use LedgerBERT in your research, please cite:
+
+```bibtex
+@article{hernandez2025dlt-corpus,
+  title={DLT-Corpus: A Large-Scale Text Collection for the Distributed Ledger Technology Domain},
+  author={Hernandez Cruz, Walter and Devine, Peter and Vadgama, Nikhil and Tasca, Paolo and Xu, Jiahua},
+  year={2025}
+}
+```
+
+## Related Resources
+
+- **DLT-Corpus Collection**: https://huggingface.co/collections/ExponentialScience/dlt-corpus-68e44e40d4e7a3bd7a224402
+- **Scientific Literature Dataset**: https://huggingface.co/datasets/ExponentialScience/DLT-Scientific-Literature
+- **Patents Dataset**: https://huggingface.co/datasets/ExponentialScience/DLT-Patents
+- **Social Media Dataset**: https://huggingface.co/datasets/ExponentialScience/DLT-Tweets
+- **Sentiment Analysis Dataset**: https://huggingface.co/datasets/ExponentialScience/DLT-Sentiment-News
+- **Fine-tuned Market Sentiment Model**: https://huggingface.co/ExponentialScience/LedgerBERT-Market-Sentiment
+
+## Model Card Contact
+
+For questions or feedback about LedgerBERT, please open an issue on the model repository or contact the authors through the DLT-Corpus GitHub repository: https://github.com/dlt-science/DLT-Corpus