初始化项目，由ModelHub XC社区提供模型

Model: HiTZ/Marmoka-es-it-Llama-3.1-8B-Instruct
Source: Original Platform

.gitattributes

@@ -0,0 +1,36 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,246 @@
+---
+license: apache-2.0
+language:
+- es
+base_model: meta-llama/Llama-3.1-8B-Instruct
+library_name: transformers
+pipeline_tag: text-generation
+datasets:
+- HiTZ/Magpie-Llama-3.1-70B-Instruct-Filtered
+- TsinghuaC3I/UltraMedical
+tags:
+- biomedical
+- medical
+- clinical
+- spanish
+- llama-3.1
+- continual-pretraining
+- merge
+widget:
+- messages:
+  - role: user
+    content: >-
+      Varón de 55 años con dolor torácico opresivo irradiado al brazo izquierdo.
+      ¿Cuál es la prueba diagnóstica inicial más adecuada?
+
+      A) Radiografía de tórax
+
+      B) Electrocardiograma de 12 derivaciones
+
+      C) Ecografía abdominal
+
+      D) Espirometría
+
+      Responde solo con la letra de la opción correcta.
+- messages:
+  - role: user
+    content: Explica el mecanismo de acción de la metformina en la diabetes tipo 2.
+- messages:
+  - role: user
+    content: >-
+      ¿Cuáles son los efectos adversos más frecuentes de los inhibidores de la
+      ECA?
+---
+
+
+<br>
+<div style="text-align: center;">
+    <img src="https://cdn-uploads.huggingface.co/production/uploads/6560b3ac1338610184656ce0/g4j2IH3ku8rJuiMa9GRz7.png" style="height: 175px;display: block;margin-left: auto;margin-right: auto;">
+</div>
+
+<h1 style="text-align: center;">
+  <b>Marmoka-es+it</b>
+<br>
+  A Lightweight Spanish Clinical Large Language Model
+<br>
+  Built on Llama 3.1
+</h1>
+
+<br>
+
+<p style="text-align: justify;">
+Marmoka is a family of lightweight 8B-parameter clinical LLMs for English and Spanish. Marmoka-es+it is a Spanish member of the family, developed to address the scarcity of high-quality medical data and instructions in Spanish. It was obtained by merging Marmoka-es, a model continually pre-trained on Spanish medical corpora, with Marmoka-it, a model fine-tuned on medical instructions, combining the strengths of both adaptation approaches. Marmoka-es+it consistently outperforms both Llama-3.1-8B-Instruct and Llama3.1-Aloe-Beta-8B on Spanish clinical multiple-choice QA, showing that robust medical LLMs can be developed for languages that remain low-resource in the medical domain.
+</p>
+
+- 📖 Paper: [To Adapt or not to Adapt, Rethinking the Value of Medical Knowledge-Aware Large Language Models](https://arxiv.org/abs/2604.06854)
+
+
+## How to Get Started with the Model
+
+You can use the model with the `transformers` pipeline (it uses the Llama 3.1 chat template):
+```python
+import torch
+from transformers import pipeline
+
+pipe = pipeline(
+    "text-generation",
+    model="HiTZ/Marmoka-es-it-Llama-3.1-8B-Instruct",
+    torch_dtype=torch.bfloat16,
+    device_map="auto",
+)
+
+messages = [
+    {"role": "system", "content": "Eres un experto médico. Responde a la pregunta de opción múltiple únicamente con la letra de la opción correcta."},
+    {"role": "user", "content": "¿Qué déficit vitamínico causa el escorbuto?\nA) Vitamina A\nB) Vitamina B12\nC) Vitamina C\nD) Vitamina D"},
+]
+
+outputs = pipe(messages, max_new_tokens=256)
+print(outputs[0]["generated_text"][-1]["content"])
+```
+
+
+# Model Description
+
+- **Developed by**: Ane G. Domingo-Aldama, Iker De La Iglesia, Maitane Urruela, Aitziber Atutxa, and Ander Barrena
+- **Contact**: [Ane G. Domingo-Aldama](mailto:ane.garciad@ehu.eus) and [Iker De La Iglesia](mailto:iker.delaiglesia@ehu.eus)
+- **Language(s) (NLP)**: Spanish
+- **License**: apache-2.0
+- **Funding**: 
+  - The Spanish Ministry of Science, Innovation and Universities, MCIN/AEI/10.13039/501100011033 projects:
+    - EDHIA PID2022-136522OB-C22 (also supported by FEDER, UE).
+    - TRUST-MED PID2025-174880OB-I00 (also supported by FEDER, UE).
+    - DeepR3 TED2021-130295B-C31 (also supported by the European Union NextGenerationEU/PRTR).
+  - Basque Government: 
+    - Research group funding IT1570-22.
+  - HiTZ Center.
+  - Predoctoral grants: FPU23/03347 (I. De La Iglesia, Spanish Ministry of Science, Innovation and Universities), PRE_2024_1_0224 (A. G. Domingo-Aldama) and PRE_2025_1_0177 (M. Urruela), Basque Government.
+
+
+## Model Details
+
+<table style="border: 1px; border-collapse: collapse;">
+    <caption>Training settings for Marmoka-es+it.</caption>
+    <tbody>
+        <tr>
+            <td>Param. no.</td>
+            <td>~8B</td>
+        </tr>
+        <tr>
+            <td>Architecture</td>
+            <td>Llama 3.1 (decoder-only)</td>
+        </tr>
+        <tr>
+            <td>Base model</td>
+            <td><a href="https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct">Llama-3.1-8B-Instruct</a></td>
+        </tr>
+        <tr>
+            <td>Adaptation</td>
+            <td>DAPT + general instructions (Marmoka-es), merged with medical instruction tuning (Marmoka-it)</td>
+        </tr>
+        <tr>
+            <td>Medical corpus</td>
+            <td>~908M (≈1B) words (Spanish)</td>
+        </tr>
+        <tr>
+            <td>General instructions</td>
+            <td>310K (Spanish, Magpie)</td>
+        </tr>
+        <tr>
+            <td>Medical instructions (Marmoka-it)</td>
+            <td>410K (UltraMedical) + 82K general-domain</td>
+        </tr>
+        <tr>
+            <td>Marmoka-es training</td>
+            <td>3 models trained with different hyperparameters on the same corpus, merged</td>
+        </tr>
+        <tr>
+            <td>Training framework</td>
+            <td><a href="https://github.com/axolotl-ai-cloud/axolotl">Axolotl</a></td>
+        </tr>
+        <tr>
+            <td>Merging tool</td>
+            <td><a href="https://github.com/arcee-ai/mergekit">MergeKit</a></td>
+        </tr>
+        <tr>
+            <td>Model selection</td>
+            <td>Casimedicos validation set (EN &amp; ES)</td>
+        </tr>
+    </tbody>
+</table>
+
+
+## Training Data
+
+<p style="text-align: justify;">
+Marmoka-es+it was built in two stages, both starting from Llama-3.1-8B-Instruct. First, Marmoka-es was trained with the hybrid strategy of Sainz et al. (2025), which combines continual domain-adaptive pretraining on Spanish medical corpora with general-domain instructions to avoid catastrophic forgetting. In parallel, Marmoka-it was trained on medical instructions from UltraMedical together with general-domain instructions, merging three models trained with different hyperparameters. Finally, both models were merged into Marmoka-es+it. Unlike traditional DAPT setups, an instruction-tuned model, rather than a base model, was used as the starting point. The medical corpora are an updated version of those gathered in De la Iglesia et al. (2025), and were processed and deduplicated using standard approaches (<a href="https://github.com/Maits27/Deduplication">deduplication code</a>).
+</p>
+
+<table style="border: 1px; border-collapse: collapse;">
+    <caption>Data sources and sizes used to train Marmoka-es+it.</caption>
+    <thead>
+        <tr>
+            <th>Type</th>
+            <th>Source</th>
+            <th>Size</th>
+        </tr>
+    </thead>
+    <tbody>
+        <tr>
+            <td rowspan="5">Medical corpora (Spanish)</td>
+            <td>WikiMed</td>
+            <td>16.3M words</td>
+        </tr>
+        <tr>
+            <td>PubMed</td>
+            <td>5.5M words</td>
+        </tr>
+        <tr>
+            <td>Medical Crawler</td>
+            <td>850.7M words</td>
+        </tr>
+        <tr>
+            <td>MeSpEn</td>
+            <td>6.9M words</td>
+        </tr>
+        <tr>
+            <td>SciELO</td>
+            <td>29M words</td>
+        </tr>
+        <tr>
+            <td>General-domain instructions</td>
+            <td><a href="https://huggingface.co/datasets/HiTZ/Magpie-Llama-3.1-70B-Instruct-Filtered">Magpie (Llama-3.1-70B-Instruct)</a></td>
+            <td>310K instructions</td>
+        </tr>
+        <tr>
+            <td rowspan="2">Marmoka-it</td>
+            <td><a href="https://huggingface.co/datasets/TsinghuaC3I/UltraMedical">UltraMedical</a> (medical)</td>
+            <td>410K instructions</td>
+        </tr>
+        <tr>
+            <td>General-domain instructions</td>
+            <td>82K instructions</td>
+        </tr>
+    </tbody>
+</table>
+
+
+
+# Limitation and Bias
+
+<p style="text-align: justify;">Marmoka-es+it is intended for research purposes. Its evaluation is based on short-form multiple-choice question answering with automated accuracy metrics and without expert clinical adjudication, so the validity of the model's underlying reasoning has not been verified, and these benchmarks may not fully capture genuine medical expertise. It is recommended to validate and test the model for specific applications before using it.</p>
+
+<p style="text-align: justify;">Instruction following and strict output formatting remain a limitation, especially in multi-step tasks such as summarization, where formatting failures were observed. Outputs should be checked whenever a strict format is required.</p>
+
+## Biases
+
+<ul>
+    <li><strong>Data Collection Bias:</strong> The training data for Marmoka-es+it was sourced from the web and open datasets, and may not adequately capture the full range of linguistic, demographic, or clinical variability found in real-world medical settings. This may introduce selection bias, and the model might reflect and propagate these biases in its outputs.</li>
+    <li><strong>Demographic and Linguistic Bias:</strong> Since the training data may not equally represent all demographic groups or linguistic nuances, the model may perform unequally across populations, potentially reinforcing existing health disparities and societal biases.</li>
+    <li><strong>Unexamined Ethical Considerations:</strong> No further assessments have been conducted to determine whether the training data contains biases or personally identifiable information, nor have the anonymization measures been independently verified. All data sources were previously processed, publicly released, and made available with stated limitations, including steps taken to ensure data quality and privacy compliance.</li>
+</ul>
+
+## Disclaimer
+<p>Marmoka-es+it is intended for research purposes and is not approved for clinical use. It must not be used as a medical device or relied upon for diagnostic or therapeutic decision-making, and its outputs should not be interpreted as medical advice. Using the model without rigorous validation poses significant risks, including misinformation, misclassification, and potential harm.</p>
+<p>When employed in actual clinical scenarios, all outputs must be independently reviewed and validated by qualified healthcare professionals. We do not take any liability for the use of this model.</p>
+
+
+# Citing information
+
+```bibtex
+@article{domingo2026adapt,
+  title={To Adapt or not to Adapt, Rethinking the Value of Medical Knowledge-Aware Large Language Models},
+  author={Domingo-Aldama, Ane G and De La Iglesia, Iker and Urruela, Maitane and Atutxa, Aitziber and Barrena, Ander},
+  journal={arXiv preprint arXiv:2604.06854},
+  year={2026}
+}
+```

config.json

@@ -0,0 +1,40 @@
+{
+  "_name_or_path": "meta-llama/Llama-3.1-8B-Instruct",
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 128000,
+  "eos_token_id": [
+    128001,
+    128008,
+    128009
+  ],
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "initializer_range": 0.02,
+  "intermediate_size": 14336,
+  "max_position_embeddings": 131072,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 32,
+  "num_hidden_layers": 32,
+  "num_key_value_heads": 8,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": {
+    "factor": 8.0,
+    "high_freq_factor": 4.0,
+    "low_freq_factor": 1.0,
+    "original_max_position_embeddings": 8192,
+    "rope_type": "llama3"
+  },
+  "rope_theta": 500000.0,
+  "tie_word_embeddings": false,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.49.0",
+  "use_cache": true,
+  "vocab_size": 128256
+}

mergekit_config.yml

@@ -0,0 +1,13 @@
+models:
+ - model: meta-llama/Llama-3.1-8B-Instruct
+ - model: /data/abarrena/marmokaPaper/marmoka-es-adv
+   parameters:
+     density: 0.85
+     weight: 0.65
+ - model: /data/abarrena/marmokaPaper/marmoka-um-adv
+   parameters:
+     density: 0.85
+     weight: 0.25
+merge_method: dare_ties
+base_model: meta-llama/Llama-3.1-8B-Instruct
+dtype: bfloat16

model-00001-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a96b31fe627e5be10609a58fb3c5e1b761a47bb54132e345c2bbce8f10dfef16
+size 4953586384

model-00002-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e7fe5bad96c91bc33d8e7d034dbf34ec5187d64fb791f610f812e551d1b5474c
+size 4999819336

model-00003-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:80339849ad06b5503de4bddc99de31be3cd06194b3786be59fcef8a54035d0f1
+size 4915916144

model-00004-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a025b5440e6abf0e6d6f5b278bbd5dbca5cbbe901ea8ce2c64d8a2c18f01876a
+size 1191234472

special_tokens_map.json

@@ -0,0 +1,16 @@
+{
+  "bos_token": {
+    "content": "<|begin_of_text|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|eot_id|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}