初始化项目，由ModelHub XC社区提供模型

Model: esa-sceva/llama3-satcom-8b
Source: Original Platform

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,167 @@
+---
+license: llama3.1
+base_model:
+- meta-llama/Llama-3.1-8B-Instruct
+tags:
+- satcom
+- satellite
+- satellite communications
+- space
+- esa
+- artes
+- llm
+- fine-tuned
+- question-answering
+language:
+- en
+model_name: esa-sceva/llama3-satcom-8b
+pipeline_tag: text-generation
+library_name: transformers
+datasets:
+- esa-sceva/satcom-qa
+- esa-sceva/satcom-mcqa
+- esa-sceva/satcom-synth-qa
+- esa-sceva/satcom-synth-qa-cot
+---
+
+# LLama3 SatCom 8B
+
+**LLama3 SatCom 8B** is a fine-tuned open Large Language Model (LLM) developed under the **ESA ARTES** programme as part of the **SatcomLLM / SCEVA (SatCom Expert Virtual Assistant)** project.  
+It is designed to support **satellite communications (SatCom)** experts, engineers, and mission planners through domain-specialised reasoning, question answering, and document-based assistance.
+
+---
+
+## Model Description
+
+- **Base model:** `meta-llama/Llama-3.1-8B-Instruct`  
+- **Fine-tuning type:** Instruction fine-tuning (IFT)  
+- **Training data:** Domain-specific question–answer datasets (manual, synthetic, and multiple-choice)  
+- **Architecture:** Decoder-only transformer, 8 billion parameters  
+- **Languages:** English  
+- **License:** LLama-3.1 Communiy License Agreement  
+
+The model has been fine-tuned on curated SatCom-related corpora to enhance its understanding of technical language, protocols, and reasoning processes common to satellite communications, including 5G/6G non-terrestrial networks, link budget evaluation, and mission engineering tasks.
+
+---
+
+## Training Datasets
+
+| Dataset                           | Description                                                                            |
+| --------------------------------- | -------------------------------------------------------------------------------------- |
+| **esa-sceva/satcom-synth-qa**     | Synthetic QA data generated via agentic pipelines using large teacher models           |
+| **esa-sceva/satcom-synth-qa-cot** | Chain-of-thought annotated QA used to improve reasoning depth and factual traceability |
+
+---
+
+
+## Intended Use
+
+**Primary use cases:**
+- Technical Q&A and reasoning on SatCom systems  
+- Support for link budget and RF engineering questions  
+- Guidance for 5G/6G NTN (Non-Terrestrial Network) operations  
+- Mission design, planning, and anomaly detection support  
+- Educational and research use within the SatCom sector  
+
+**Intended users:**  
+- ESA engineers and project officers  
+- SatCom and aerospace researchers  
+- SMEs and technical operators in satellite communication  
+- Academic and educational users  
+
+---
+
+## Limitations
+
+- The model does **not access real-time mission data** or proprietary ESA documents.  
+- Answers are based on training data and may require expert validation for operational use.  
+- It should not be relied upon for flight-critical or safety-critical decisions.  
+- Limited context window (base 8B configuration) may constrain long-document reasoning.
+
+---
+
+## Technical Details
+
+| Parameter | Value |
+|------------|--------|
+| Base Model | Llama 3.1 8B Instruct |
+| Parameters | 8 billion |
+| Context length | 8k tokens |
+| Precision | bfloat16 / fp16 |
+| Framework | Lit-GPT (Lightning AI) |
+| Training infra | EuroHPC MareNostrum5 + AWS EC2 |
+| Optimisation | LoRA fine-tuning, cosine LR schedule |
+
+---
+
+## Evaluation
+
+### Evaluation Datasets
+
+The model was benchmarked on both general-purpose and domain-specific QA tasks. Regarding Satcom-specific datasets:
+
+| Dataset                   | Subset             | Description                                                                                          |
+| ------------------------- | ------------------ | ---------------------------------------------------------------------------------------------------- |
+| **esa-sceva/satcom-qa**   | **Open SatCom QA** | Conceptual and reasoning-based questions on SatCom workflows, regulations, and mission/system design |
+|                           | **Math SatCom QA** | Quantitative and formula-based questions derived from system design and orbital mechanics topics     |
+| **esa-sceva/satcom-mcqa** | **Open MCQA**      | Conceptual multiple-choice questions on RF systems, communication protocols, and architecture        |
+|                           | **Math MCQA**      | Numerical and link-budget-focused multiple-choice questions testing applied calculations             |
+
+### Results
+
+<table>
+  <thead>
+    <tr>
+      <th rowspan="2">Model</th>
+      <th colspan="3">MCQA (Accuracy)</th>
+      <th colspan="3">Satcom-QA</th>
+      <th colspan="3">EVE-QA</th>
+    </tr>
+    <tr>
+      <th>Satcom</th>
+      <th>EVE</th>
+      <th>TeleQnA</th>
+      <th>Norm.</th>
+      <th>Bin.</th>
+      <th>WR</th>
+      <th>Norm.</th>
+      <th>Bin.</th>
+      <th>WR</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td>Llama-3.1-8B-Instruct</td>
+      <td>78.59</td>
+      <td><strong>81.35</strong></td>
+      <td>68.40</td>
+      <td>75.44</td>
+      <td>70.40</td>
+      <td>—</td>
+      <td>61.47</td>
+      <td>65.20</td>
+      <td>—</td>
+    </tr>
+    <tr style="border-top: 1px dashed #999;">
+      <td><code>llama3-satcom-8b</code></td>
+      <td><strong>80.15</strong></td>
+      <td>80.95</td>
+      <td><strong>68.80</strong></td>
+      <td><strong>77.75</strong></td>
+      <td><strong>73.62</strong></td>
+      <td><strong>51.49</strong></td>
+      <td><strong>62.90</strong></td>
+      <td><strong>68.12</strong></td>
+      <td><strong>51.41</strong></td>
+    </tr>
+  </tbody>
+</table>
+
+**Table:** Evaluation results (%).  
+ - **Norm.** denotes the normalized score obtained by averaging 1–5 ratings from a panel of LLM judges (Qwen3, gpt-4.1-mini, Mistral-Large-2512, and DeepSeek-V3.2) and scaling to [0,1].  
+ - **Bin.** is the binary accuracy computed from correctness judgments.  
+ - **WR** (Adjusted Win Rate) is defined as (wins + 0.5 × ties) / total, based on pairwise comparisons with randomized answer order.  
+Multiple-choice performance is measured using standard accuracy.  
+All results are averaged over 3–5 runs; standard deviation ≤ 0.25 pp for open-ended QA and ≤ 0.10 pp for MCQA.
+
+---

config.json

@@ -0,0 +1,38 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 128000,
+  "eos_token_id": [
+    128001,
+    128008,
+    128009
+  ],
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "initializer_range": 0.02,
+  "intermediate_size": 14336,
+  "max_position_embeddings": 131072,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 32,
+  "num_hidden_layers": 32,
+  "num_key_value_heads": 8,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": {
+    "factor": 8.0,
+    "low_freq_factor": 1.0,
+    "high_freq_factor": 4.0,
+    "original_max_position_embeddings": 8192,
+    "rope_type": "llama3"
+  },
+  "rope_theta": 500000.0,
+  "tie_word_embeddings": false,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.42.3",
+  "use_cache": true,
+  "vocab_size": 128256
+}

generation_config.json

@@ -0,0 +1,12 @@
+{
+  "bos_token_id": 128000,
+  "do_sample": true,
+  "eos_token_id": [
+    128001,
+    128008,
+    128009
+  ],
+  "temperature": 0.6,
+  "top_p": 0.9,
+  "transformers_version": "4.42.3"
+}

hyperparameters.yaml

@@ -0,0 +1,51 @@
+checkpoint_dir: /teamspace/studios/this_studio/out/finetune/llama-8b-dataset12/final
+out_dir: out/finetune-2/llama-8b-dataset12-cot
+precision: bf16-true
+devices: 1
+num_nodes: 1
+lora_r: 8
+lora_alpha: 16
+lora_dropout: 0.1
+lora_query: true
+lora_key: false
+lora_value: true
+lora_projection: false
+lora_mlp: false
+lora_head: false
+data:
+  class_path: litgpt.data.JSON
+  init_args:
+    json_path: cot_satcom_litgpt.json
+    mask_prompt: false
+    val_split_fraction: 0.05
+    prompt_style: alpaca
+    ignore_index: -100
+    seed: 42
+    num_workers: 4
+train:
+  save_interval: 25
+  log_interval: 1
+  global_batch_size: 32
+  micro_batch_size: 4
+  lr_warmup_steps: 25
+  epochs: 12
+  max_seq_length: 2048
+  min_lr: 2.0e-05
+log: {}
+eval:
+  interval: 20
+  max_new_tokens: 100
+  max_iters: 100
+  initial_validation: true
+  final_validation: true
+  evaluate_example: first
+optimizer:
+  class_path: torch.optim.AdamW
+  init_args:
+    lr: 2.0e-05
+    weight_decay: 0.0
+    betas:
+    - 0.9
+    - 0.999
+logger_name: wandb
+seed: 1337

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:42c96afe2c1e14c61175cddd644c6005c121f8ff60855e28455f934cd9114098
+size 16060556616

model_config.yaml

@@ -0,0 +1,49 @@
+attention_logit_softcapping: null
+attention_scores_scalar: null
+attn_bias: false
+bias: false
+block_size: 131072
+final_logit_softcapping: null
+gelu_approximate: none
+head_size: 128
+hf_config:
+  name: Meta-Llama-3.1-8B-Instruct
+  org: meta-llama
+intermediate_size: 14336
+latent_attention: null
+lm_head_bias: false
+mlp_class_name: LLaMAMLP
+moe_intermediate_size: null
+n_embd: 4096
+n_expert: 0
+n_expert_per_token: 0
+n_head: 32
+n_layer: 32
+n_query_groups: 8
+name: Llama-3.1-8B-Instruct
+norm_1: true
+norm_2: true
+norm_class_name: RMSNorm
+norm_eps: 1.0e-05
+norm_qk: false
+norm_qk_type: default
+padded_vocab_size: 128256
+padding_multiple: 512
+parallel_residual: false
+post_attention_norm: false
+post_mlp_norm: false
+rope_adjustments:
+  factor: 8.0
+  high_freq_factor: 4.0
+  low_freq_factor: 1.0
+  original_max_seq_len: 8192
+rope_base: 500000
+rope_condense_ratio: 1
+rope_indices: null
+rope_local_base_freq: null
+rotary_percentage: 1.0
+scale_embeddings: false
+shared_attention_norm: false
+sliding_window_indices: null
+sliding_window_size: null
+vocab_size: 128000