---
license: mit
language:
- ko
base_model:
- K-intelligence/Midm-2.0-Base-Instruct
tags:
- Korean
- Culture
---


# Midm-KCulture-2.0-Base-Instruct
- This model is fine-tuned from KT/Midm-2.0-Base-Instruct on the 'Korean Culture Q&A Corpus' using the LoRA (Low-Rank Adaptation) methodology.

## GitHub
Check out the full training code [here](https://github.com/dahlia52/KR-Culture-QA/tree/main).

## Training Hyperparameters

| Hyperparameter                | Value                         |
| :---------------------------- | :---------------------------- |
| **SFTConfig** |                               |
| `torch_dtype`                 | `bfloat16`                    |
| `seed`                        | `42`                          |
| `epoch`                       | `3`                           |
| `per_device_train_batch_size` | `2`                           |
| `per_device_eval_batch_size`  | `2`                           |
| `learning_rate`               | `0.0002`                      |
| `lr_scheduler_type`           | `"linear"`                    |
| `max_grad_norm`               | `1.0`                         |
| `neftune_noise_alpha`         | `None`                        |
| `gradient_accumulation_steps` | `1`                           |
| `gradient_checkpointing`      | `False`                       |
| `max_seq_length`              | `1024`                        |
| **LoraConfig** |                               |
| `r`                           | `16`                          |
| `lora_alpha`                  | `16`                          |
| `lora_dropout`                | `0.1`                         |
| `target_modules`              | `["q_proj", "v_proj"]`        |

## Usage
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
model_name = "jjae/Midm-KCulture-2.0-Base-Instruct"
model = AutoModelForCausalLM.from_pretrained(
      model_name,
      torch_dtype=torch.bfloat16,
      trust_remote_code=True,
      device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(model_name)
```