tmfi-us/Progenitor-V5-Final-LLaMa-70B

Update Feb 16, 2025 noon PST:

Upon debugging with the author of the original model, the author decided to redo the model as something is off with the weights. See discussion here.

NOTE: This means this repo's weights are also broken!

Update Feb 16, 2025 morning PST:

The author of the original model mentioned that this model gave very different outputs. See ongoing discussion here.

Overview

The original model had invalid tensor.Shape for weights ([1, 8192]), raising following errors when loading with transformers:

ValueError: Trying to set a tensor of shape torch.Size([1, 8192]) in "weight" (which has shape torch. Size ( [8192])), this looks incorrect.

So I resized them into [8192] with following script:

import os
from safetensors.torch import load_file, save_file

# Update this to point to your safetensors directory
MODEL_DIR = "/root/.cache/huggingface/hub/models--Tarek07--Progenitor-V5-Final-LLaMa-70B/snapshots/8ca900fd3a65a725902d525e518be1bf374c0247"
DEST_DIR = "/output/Progenitor-V5-Final-LLaMa-70B"

def fix_shard(shard_path, output_path):
    # Load the shard
    data = load_file(shard_path)
    # data is a dict:  key -> torch.Tensor

    # Go through every tensor and fix the shape if necessary
    for key, tensor in data.items():
        # Check if the shape is (1, 8192) instead of (8192)
        if list(tensor.shape) == [1, 8192]:
            print(f"  Fixing {key} in {os.path.basename(shard_path)} from {tensor.shape} to (8192,)")
            # Either squeeze(0) or view(-1) or view(8192):
            #   data[key] = tensor.squeeze(0)
            # or
            data[key] = tensor.view(8192)

    # Save the fixed shard to output_path
    save_file(data, output_path, metadata={"format": "pt"})
    print(f"  -> Saved fixed shard to: {output_path}")

def main():
    # Look for .safetensors files in MODEL_DIR
    for filename in sorted(os.listdir(MODEL_DIR)):
        if filename.endswith(".safetensors"):
            shard_path = os.path.join(MODEL_DIR, filename)
            output_path = os.path.join(DEST_DIR, f"{filename}")

            print(f"Processing: {shard_path}")
            fix_shard(shard_path, output_path)

if __name__ == "__main__":
    main()

Original README.md from here:

This marks the culmination of my experiments with the Progenitor series. I fixed the typo I had earlier where it wasn't computing in float32, but 6 models in computed in float32 is a bit taxing on resources and time and so I left it for the configuration I thought was the best (it's not something I can afford to do with every model I make, just the worthwhile ones). This one also uses the Sicari's tokenizer which I find the best.

merge

This is a merge of pre-trained language models created using mergekit.

Merge Details

Merge Method

This model was merged using the Linear DELLA merge method using meta-llama/Llama-3.3-70B-Instruct as a base.

Models Merged

The following models were included in the merge:

• TheDrummer/Anubis-70B-v1
• EVA-UNIT-01/EVA-LLaMA-3.33-70B-v0.1
• Sao10K/70B-L3.3-Cirrus-x1
• SicariusSicariiStuff/Negative_LLAMA_70B
• Sao10K/L3.1-70B-Hanami-x1

Configuration

The following YAML configuration was used to produce this model:

models:
  - model: Sao10K/L3.1-70B-Hanami-x1
    parameters:
      weight: 0.20
      density: 0.7
  - model: Sao10K/70B-L3.3-Cirrus-x1
    parameters:
      weight: 0.20
      density: 0.7
  - model: SicariusSicariiStuff/Negative_LLAMA_70B
    parameters:
      weight: 0.20
      density: 0.7
  - model: TheDrummer/Anubis-70B-v1
    parameters:
      weight: 0.20
      density: 0.7
  - model: EVA-UNIT-01/EVA-LLaMA-3.33-70B-v0.1
    parameters:
      weight: 0.20
      density: 0.7
merge_method: della_linear
base_model: meta-llama/Llama-3.3-70B-Instruct
parameters:
  epsilon: 0.2
  lambda: 1.1
  int8_mask: true
dtype: float32
out_dtype: bfloat16
tokenizer:
 source: SicariusSicariiStuff/Negative_LLAMA_70B