初始化项目，由ModelHub XC社区提供模型

Model: QuantFactory/Romulus-cpt-Llama-3.1-8B-v0.1-GGUF
Source: Original Platform

.gitattributes

@@ -0,0 +1,48 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+Romulus-cpt-Llama-3.1-8B-v0.1.Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+Romulus-cpt-Llama-3.1-8B-v0.1.Q4_1.gguf filter=lfs diff=lfs merge=lfs -text
+Romulus-cpt-Llama-3.1-8B-v0.1.Q4_0.gguf filter=lfs diff=lfs merge=lfs -text
+Romulus-cpt-Llama-3.1-8B-v0.1.Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
+Romulus-cpt-Llama-3.1-8B-v0.1.Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
+Romulus-cpt-Llama-3.1-8B-v0.1.Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+Romulus-cpt-Llama-3.1-8B-v0.1.Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+Romulus-cpt-Llama-3.1-8B-v0.1.Q5_0.gguf filter=lfs diff=lfs merge=lfs -text
+Romulus-cpt-Llama-3.1-8B-v0.1.Q5_1.gguf filter=lfs diff=lfs merge=lfs -text
+Romulus-cpt-Llama-3.1-8B-v0.1.Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
+Romulus-cpt-Llama-3.1-8B-v0.1.Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
+Romulus-cpt-Llama-3.1-8B-v0.1.Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+Romulus-cpt-Llama-3.1-8B-v0.1.Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,233 @@
+
+---
+
+datasets:
+- louisbrulenaudet/Romulus-cpt-fr
+license: llama3
+language:
+- fr
+base_model: meta-llama/Meta-Llama-3.1-8B-Instruct
+pipeline_tag: text-generation
+library_name: transformers
+tags:
+- law
+- droit
+- unsloth
+- trl
+- transformers
+- sft
+- llama
+
+---
+
+[![QuantFactory Banner](https://lh7-rt.googleusercontent.com/docsz/AD_4nXeiuCm7c8lEwEJuRey9kiVZsRn2W-b4pWlu3-X534V3YmVuVc2ZL-NXg2RkzSOOS2JXGHutDuyyNAUtdJI65jGTo8jT9Y99tMi4H4MqL44Uc5QKG77B0d6-JfIkZHFaUA71-RtjyYZWVIhqsNZcx8-OMaA?key=xt3VSDoCbmTY7o-cwwOFwQ)](https://hf.co/QuantFactory)
+
+
+# QuantFactory/Romulus-cpt-Llama-3.1-8B-v0.1-GGUF
+This is quantized version of [louisbrulenaudet/Romulus-cpt-Llama-3.1-8B-v0.1](https://huggingface.co/louisbrulenaudet/Romulus-cpt-Llama-3.1-8B-v0.1) created using llama.cpp
+
+# Original Model Card
+
+<img src="assets/thumbnail.webp">
+
+# Romulus, continually pre-trained models for French law.
+
+Romulus is a series of continually pre-trained models enriched in French law and intended to serve as the basis for a fine-tuning process on labeled data. Please note that these models have not been aligned for the production of usable text as they stand, and will certainly need to be fine-tuned for the desired tasks in order to produce satisfactory results.
+
+The training corpus is made up of around 34,864,949 tokens (calculated with the meta-llama/Meta-Llama-3.1-8B tokenizer).
+
+## Hyperparameters
+
+The following table outlines the key hyperparameters used for training Romulus.
+
+| **Parameter**                   | **Description**                                                 | **Value**                   |
+|----------------------------------|-----------------------------------------------------------------|-----------------------------|
+| `max_seq_length`                 | Maximum sequence length for the model                           | 4096                        |
+| `load_in_4bit`                   | Whether to load the model in 4-bit precision                    | False                       |
+| `model_name`                     | Pre-trained model name from Hugging Face                        | meta-llama/Meta-Llama-3.1-8B|
+| `r`                              | Rank of the LoRA adapter                                        | 128                         |
+| `lora_alpha`                     | Alpha value for the LoRA module                                 | 32                          |
+| `lora_dropout`                   | Dropout rate for LoRA layers                                    | 0                           |
+| `bias`                           | Bias type for LoRA adapters                                     | none                        |
+| `use_gradient_checkpointing`     | Whether to use gradient checkpointing                           | unsloth                     |
+| `train_batch_size`               | Per device training batch size                                  | 8                           |
+| `gradient_accumulation_steps`    | Number of gradient accumulation steps                           | 8                           |
+| `warmup_ratio`                   | Warmup steps as a fraction of total steps                       | 0.1                         |
+| `num_train_epochs`               | Number of training epochs                                       | 1                           |
+| `learning_rate`                  | Learning rate for the model                                     | 5e-5                        |
+| `embedding_learning_rate`        | Learning rate for embeddings                                    | 1e-5                        |
+| `optim`                          | Optimizer used for training                                     | adamw_8bit                  |
+| `weight_decay`                   | Weight decay to prevent overfitting                             | 0.01                        |
+| `lr_scheduler_type`              | Type of learning rate scheduler                                 | linear                      |
+
+# Training script
+
+Romulus was trained using Unsloth on a Nvidia H100 Azure EST US instance provided by the Microsoft for Startups program from this script:
+
+```python
+# -*- coding: utf-8 -*-
+import os
+
+from typing import (
+    Dict,
+)
+
+from datasets import load_dataset
+from unsloth import (
+    FastLanguageModel,
+    is_bfloat16_supported,
+    UnslothTrainer,
+    UnslothTrainingArguments,
+)
+
+max_seq_length = 4096
+dtype = None
+load_in_4bit = False
+
+model, tokenizer = FastLanguageModel.from_pretrained(
+    model_name="meta-llama/Meta-Llama-3.1-8B",
+    max_seq_length=max_seq_length,
+    dtype=dtype,
+    load_in_4bit=load_in_4bit,
+    token="hf_token",
+)
+
+model = FastLanguageModel.get_peft_model(
+    model,
+    r=128,
+    target_modules=[
+        "q_proj",
+        "k_proj",
+        "v_proj",
+        "o_proj",
+        "gate_proj",
+        "up_proj",
+        "down_proj",
+        "embed_tokens",
+        "lm_head",
+    ],
+    lora_alpha=32,
+    lora_dropout=0,
+    bias="none",
+    use_gradient_checkpointing="unsloth",
+    random_state=3407,
+    use_rslora=True,
+    loftq_config=None,
+)
+
+prompt = """### Référence :
+{}
+### Contenu :
+{}"""
+
+EOS_TOKEN = tokenizer.eos_token
+
+def formatting_prompts_func(examples):
+    """
+    Format input examples into prompts for a language model.
+
+    This function takes a dictionary of examples containing titles and texts,
+    combines them into formatted prompts, and appends an end-of-sequence token.
+
+    Parameters
+    ----------
+    examples : dict
+        A dictionary containing two keys:
+        - 'title': A list of titles.
+        - 'text': A list of corresponding text content.
+
+    Returns
+    -------
+    dict
+        A dictionary with a single key 'text', containing a list of formatted prompts.
+
+    Notes
+    -----
+    - The function assumes the existence of a global `prompt` variable, which is a
+      formatting string used to combine the title and text.
+    - The function also assumes the existence of a global `EOS_TOKEN` variable,
+      which is appended to the end of each formatted prompt.
+    - The input lists 'title' and 'text' are expected to have the same length.
+
+    Examples
+    --------
+    >>> examples = {
+    ...     'title': ['Title 1', 'Title 2'],
+    ...     'text': ['Content 1', 'Content 2']
+    ... }
+    >>> formatting_cpt_prompts_func(examples)
+    {'text': ['<formatted_prompt_1><EOS>', '<formatted_prompt_2><EOS>']}
+    """
+    refs = examples["ref"]
+    texts = examples["texte"]
+    outputs = []
+
+    for ref, text in zip(refs, texts):
+        text = prompt.format(ref, text) + EOS_TOKEN
+        outputs.append(text)
+
+    return {
+        "text": outputs,
+    }
+
+
+cpt_dataset = load_dataset(
+    "louisbrulenaudet/Romulus-cpt-fr",
+    split="train",
+    token="hf_token",
+)
+
+cpt_dataset = cpt_dataset.map(
+    formatting_prompts_func,
+    batched=True,
+)
+
+trainer = UnslothTrainer(
+    model=model,
+    tokenizer=tokenizer,
+    train_dataset=cpt_dataset,
+    dataset_text_field="text",
+    max_seq_length=max_seq_length,
+    dataset_num_proc=2,
+    args=UnslothTrainingArguments(
+        per_device_train_batch_size=8,
+        gradient_accumulation_steps=8,
+        warmup_ratio=0.1,
+        num_train_epochs=1,
+        learning_rate=5e-5,
+        embedding_learning_rate=1e-5,
+        fp16=not is_bfloat16_supported(),
+        bf16=is_bfloat16_supported(),
+        logging_steps=1,
+        report_to="wandb",
+        save_steps=350,
+        run_name="romulus-cpt",
+        optim="adamw_8bit",
+        weight_decay=0.01,
+        lr_scheduler_type="linear",
+        seed=3407,
+        output_dir="outputs",
+    ),
+)
+
+trainer_stats = trainer.train()
+```
+
+<img src="assets/loss.png">
+
+## Citing & Authors
+
+If you use this code in your research, please use the following BibTeX entry.
+
+```BibTeX
+@misc{louisbrulenaudet2024,
+  author =       {Louis Brulé Naudet},
+  title =        {Romulus, continually pre-trained models for French law},
+  year =         {2024}
+  howpublished = {\url{https://huggingface.co/datasets/louisbrulenaudet/Romulus-cpt-fr}},
+}
+```
+
+## Feedback
+
+If you have any feedback, please reach out at [louisbrulenaudet@icloud.com](mailto:louisbrulenaudet@icloud.com).

Romulus-cpt-Llama-3.1-8B-v0.1.Q2_K.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:880be810d284a97535dd643e86b1ca124e5a2db3daf0420d185c27acac07ebc5
+size 3179131808

Romulus-cpt-Llama-3.1-8B-v0.1.Q3_K_L.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4d7be343f2287e51cb9e4e31cbc86daa5e109da037beca5402c04f125d94b6d8
+size 4321956768

Romulus-cpt-Llama-3.1-8B-v0.1.Q3_K_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2d33ae564db7b92cf8407c953fac7b2c503adc780bd43b213cfcbf205f5c00f4
+size 4018918304

Romulus-cpt-Llama-3.1-8B-v0.1.Q3_K_S.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c2feee9d82c4086f98cfdc6642cd1b6a22df9771bbb92ca1807b0d8b37da1740
+size 3664499616

Romulus-cpt-Llama-3.1-8B-v0.1.Q4_0.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d7714e95154639f3595cc2aa7c8b6a74fd4c723607bf3570b9ff80db43f1170f
+size 4661212064

Romulus-cpt-Llama-3.1-8B-v0.1.Q4_1.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2e2cc31be0429cf5f00255269fea7f4c7691852dea3b94a1a4ffbd50071f4ac0
+size 5130253216

Romulus-cpt-Llama-3.1-8B-v0.1.Q4_K_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9339ba97c7f21b1d5ae6e4a3276c73664a2d77f12f599171b2cdf02330942898
+size 4920734624

Romulus-cpt-Llama-3.1-8B-v0.1.Q5_0.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:46232eda88360af13fd6e3ea46ab004c235be5a10f92785981574fddab995c48
+size 5599294368

Romulus-cpt-Llama-3.1-8B-v0.1.Q5_1.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8a7278e1a9dd2c76cc26f0ee162fcc6cb73a4ea18d9f321fcab4330870a7bd5c
+size 6068335520

Romulus-cpt-Llama-3.1-8B-v0.1.Q5_K_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a3c98da0610a325a267263d1cb14e3f0ffa0f1ee84081a18b8f1378941ff7c3f
+size 5732987808

Romulus-cpt-Llama-3.1-8B-v0.1.Q5_K_S.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:61e4d895ebf54fae22c0d634c08c517ed6f0e34ee6ad0fa3ab4776aef710269a
+size 5599294368

Romulus-cpt-Llama-3.1-8B-v0.1.Q6_K.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:59d2c4023d20e88d00a2a97cb10cbe5d7e6b4ecbb8883f49c239d841410827ab
+size 6596006816

Romulus-cpt-Llama-3.1-8B-v0.1.Q8_0.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:62ba5c081365fa5386cdcad9357ca0d7826bcb0e648b74436e67a802d2b279db
+size 8540771232

configuration.json

@@ -0,0 +1 @@
+{"framework": "pytorch", "task": "text-generation", "allow_remote": true}